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1.0  EXECUTIVE  SUMMARY 

1.1  INTRODUCTION  TO  THE  LEVEL  I  ERGONOMICS 
ASSESSMENT  AND  PROBLEM-SOLVING  METHODOLOGY 
RESEARCH  REPORT 

The  U.S.  Air  Force  has  sponsored  the  development  of  standard  ergonomics  assessment 
methodology  guides  and  management  tools  which  will  be  integrated  into  the  AFOSH 
Program.  These  methodologies  and  tools  will  be  used  as  a  means  to  minimize  or 
eliminate  work-related  musculoskeletal  disorders  (WMDs)  associated  with  routine 
exposure  to  ergonomics  risk  factors  at  Air  Force  installations. 

This  Research  Report  describes  how  effectively  the  Level  I  Ergonomics  Assessment  and 
Problem-Solving  Methodology  for  Maintenance  and  Inspection  Work  Areas  performed  in 
testing,  and  therefore,  how  effective  it  will  be  as  a  tool  for  providing  the  means  for 
Bioenvironmental  Engineering  (BEF)  to  conduct  aggressive  task-based  problem-solving 
in  an  Ergonomics  Problem  Area  (EPRA).  The  methodology  which  this  report  describes 
was  initially  used  for  the  development  and  testing  of  the  Administrative  Guide.  It  will 
subsequently  be  applied  to  the  development  and  testing  of  the  Warehouse  and  Assembly 
Guides.  In  order  to  obtain  a  clear  understanding  of  the  results,  an  overview  of  the 
Methodology  design  and  development  process  is  provided  below. 

1.2  METHODOLOGY  DESIGN 

The  Methodology,  designed  for  use  by  a  BEF  technician  with  2-3  years  of  experience,  is 
applied  by  following  five  primary  steps: 

Step  1.  Preparation 

Step  2.  Risk  Factor  Identification 

Step  3 .  Prioritization  of  Hazards 

Step  4.  Hazard  Control 

Step  5.  Recommendations 

The  Level  I  Ergonomics  Assessment  process  is  shown  in  Figure  1.1. 
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Figure  1.1 

Level  1  Assessment  Process 


1. 


2. 


3. 


4. 


5. 


TOOL  REQUIRED 


1.3  DEVELOPMENT  AND  TESTING  PROCESS 

The  above  design  is  the  result  of  an  iterative  development  and  testing  process  that 
benefited  fi-om  the  support  and  cooperation  of  Air  Force  personnel  at  several  Air  Force 
Materiel  Command  (AFMC)  locations: 

•  Armstrong  Laboratory/Brooks  AFB,  Texas 

•  Wright-Patterson  AFB,  Ohio 

•  Eglin  AFB,  Florida 

•  Tinker  AFB,  Oklahoma 

•  Hill  AFB,  Utah 

The  Methodology  design  incorporates  the  results  of  a  focused  literature  review,  strategic 
site  visits,  criteria  established  by  the  Air  Force,  and  input  fi-om  Air  Force  technical 
advisors. 
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The  testing  and  validation  processes  were  conducted  in  two  primary  phases:  alpha  testing 
and  beta  testing.  During  the  alpha  phase,  ergonomists  tested  the  Methodology.  During 
the  beta  phase.  Air  Force  personnel  tested  the  Methodology.  For  each  phase,  the 
Methodology  was  tested  for  usability,  reliability,  sensitivity,  and  validity.  Usability 
testing  was  performed  to  ensure  that  users  would  be  able  to  apply  the  Methodology  as 
intended.  Reliability  testing  was  performed  to  determine  how  consistently  that 
application  of  the  Methodology  yielded  the  same  results.  Sensitivity  testing  was 
performed  to  determine  if  the  Level  I  Assessment  can  determine  the  difference  between 
actual  risk  levels  in  a  job.  Finally,  validity  testing  was  conducted  to  measure  how  closely 
the  results  from  experienced  ergonomists  matched  the  results  obtained  by  Air  Force 
personnel.  The  methods,  rationale,  and  results  of  the  testing  and  validation  are  detailed  in 
this  report. 

1.4  PROJECT  RESULTS 

The  results  of  the  validation  process  provide  evidence  of  the  validity,  reliability, 
sensitivity,  and  practicality  of  the  Methodology.  The  results  are  summarized  below: 

VaUdity 

•  All  of  the  job  factor  questions  were  supported  by  scientific  research.  The  list  of 
job  factor  questions  was  inclusive  of  the  Job  Factors  which  would  occur  in 
maintenance/inspection  tasks. 

•  The  overall  theoretical  framework  of  the  checklist  was  a  logical  structure  used  in 
assessment  tools  which  have  been  validated. 

•  The  general  class  of  corrective  actions  selected  by  the  end-user  agreed  with  the 
classes  selected  by  a  consensus  of  ergonomists  61%  to  70%  of  the  time.  This 
result  suggested  that  the  Level  I  Ergonomics  Assessment  assisted  end-users  in 
generating  solutions  that  experts  would  recommend. 

•  There  was  a  significant  relationship  between  the  assessment  results  of  end-users 
and  the  Gold  Standard  ergonomists. 
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Reliability 


•  The  end  users  agreed  on  the  selection  of  Job  Factors  67%  percent  of  the  time. 
Sensitivity 

•  The  Methodology  was  sensitive  enough  to  significantly  distinguish  between 
different  levels  of  exposure  and  ergonomic  risk  factors  (e.g.,  high  and  medium 
risk). 

Practicality  and  Usability 

•  The  Methodology  received  favorable  usability  comments  and  was  well  accepted 
by  BEF  technicians; 

•  The  Methodology  met  and  improved  on  the  “time  for  completion”  requirements 
established  by  the  Air  Force; 

•  The  Methodology  allowed  end  users  to  elicit  solutions  with  a  high  level  of 
reliability;  and 

•  The  Methodology  was  effective  on  a  variety  of  Air  Force  Maintenance  & 
Inspection  tasks. 

The  Level  I  Methodology  for  Maintenance  and  Inspection  Work  Areas  can  be  considered 
an  effective  means  for  identifying  and  controlling  ergonomics  hazards  in  Maintenance  & 
Inspection  work  areas.  In  addition,  it  represents  a  strong  foimdation  for  developing 
similar  approaches  to  reducing  or  eliminating  the  potential  for  WMDs  in  Warehouse  and 
Assembly  Work  Areas. 

Overall,  the  Methodology  will  enable  Air  Force  persoimel  to  identify  appropriate 
solutions  to  the  ergonomics  hazards  identified  in  Maintenance  &  Inspection  jobs.  The 
specific  objectives  which  served  as  the  basis  for  the  Methodology  development  are 
compared  with  the  actual  performance  of  the  Methodology  in  Table  1.1. 
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Table  1.1 

Summary  of  Objectives  and  Performance 


Design 

Feature 

Air  Force 
Criteria/Objective 

Actuai  Methodology  Performance 

Meets  needs  of 
end  user 

Design  the  Methodology  for  use 
by  a  BEF  technician  wiA  2-3 
years  of  experience. 

Objective  met 

•  Usability  comments  obtained  during  and  after 
beta  testing  indicate  that  the  Methodology  design 
is  appropriate  to  the  expertise  of  the  intended 
user. 

•  The  use  of  technical  “jargon”  is  avoided. 

•  Clear  definitions  are  provided  to  eliminate  the 
need  for  risk  factor  interpretation. 

•  Step-by-step  instructions  are  provided  to  the 
Methodology  user  through  the  entire  analysis  and 
pattern-matching  process. 

Ease  of  data 
collection  analysis 

Design  the  Methodology  to 
enable  the  user  to  complete  the 
data  collection  and  analysis  on  a 
Maintenance  &  Inspection  job  in 
1-2  hours.  The  goal  was  to 
minimize  time  requirements. 

Objective  exceeded. 

•  In  trials,  the  average  time  required  by  BEF 
technicians  to  complete  the  analysis  process  using 
the  Level  I  Ergonomics  Assessment  Checklist  and 
Checklist  Scoring  Summary  was  approximately 

13  minutes. 

•  Data  collection  is  anticipated  to  require  30  and  60 
minutes. 

Ease  of  data 
collection 

The  Methodology  can  be 
completed  through  visual 
observation  and 
employee/supervisor  interview. 

Objective  met 

•  Observing  the  job/tasks  is  the  primary  means  for 
collecting  data. 

•  Employees  are  interviewed  informally  to  obtain 
comments  that  may  be  useful  in  the  hazard 
identification  or  improvement  processes. 

Ease  of  problem 
identification 

The  Methodology  should  enable 
the  user  to  identify  potentially 
hazardous  tasks  within  a  shop 
and  job. 

Objective  met. 

•  The  Level  I  Ergonomics  Assessment  Checklist 
and  Checklist  Scoring  Summary  provides  an 
Overall  Job  Priority  score  as  well  as  Task  Priority 
Score(s). 

-  The  Overall  Job  Priority  Score  enables  the 
shop  to  determine  which  jobs  to  fix  first. 

-  The  Task  Score(s)  enables  a  corrective  action 
team  to  focus  on  the  specific  part  of  the  Job 
which  contributes  most  significantly  to 
ergonomics  hazard  exposure. 
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Table  1.1 

Summary  of  Objectives  and  Performance  (Cont’d) 


Design 

Feature 

Air  Force 
Criteria/Objective 

Actual  Methodology  Performance 

Ease  of  analysis 

Enable  the  user  to  determine  if 
the  content  of  the  job  and  task(s) 
meet  established  ergonomics 
(risk  factor  exposure)  criteria. 

Objective  met. 

•  Risk  factor  exposure  time  is  calculated 
automatically  (the  user  simply  circles  the 
appropriate  response):  both  the  total  task 
duration  as  well  as  the  proportion  of  the  task  time 
in  which  the  Job  Factor  occurs  are  considered. 

•  The  scoring  process  provides  a  “high”, 

“medium”,  or  “low”  priority  score  for  both  the 
Overall  Job  and  the  individual  tasks  within  the 
job. 

Ease  of  analysis 

The  Methodology  will 
incoiporate  lessons  learned  from 
the  Job  Requirements  and 

Physical  Demands  (JR/PD 

Survey)  developed  for  use  by 
Public  Health  (PHF). 

Objective  met 

•  The  User’s  Methodology  describes  how  to  make 
the  best  use  of  information  obtained  from  the 
Survey. 

•  Information  obtained  from  Part  III  of  the  JR/PD 
Survey  provides  an  initial  basis  for  Job  Selection. 

•  The  50  Case  Study  Problem-Solving  Matrices 
were  developed  to  correspond  with  the 
Maintenance  &  Inspection  task  types  listed  in  Part 
III  of  the  survey. 

•  The  identical  body  region  categories  are  used  in 
the  survey  and  in  the  Level  I  Ergonomics 
Assessment  Checklist. 

•  The  format  and  style  of  the  documents  are 
consistent. 

Ease  of  control 
identification 

Provide  Case  Studies  that  will 
serve  as  the  basis  for  the  pattern- 
matching  process  used  to  match 
the  hazards  identified  in  the 
tasks  with  controls  that  will 
reduce  employee  exposure  to 
those  hazards. 

Objective  exceeded. 

•  In  addition  to  50  case  studies,  a  section  of  the 
Methodology  Guide  is  devoted  to  design  criteria 
for  implementing  the  controls  that  are  most 
appropriate  for  the  job. 

■  Ease  of  control 
:  identification 

Design  the  Methodology  to 
enable  the  user  to  identify 
controls  and  develop  a  summary 
report  in  1-2  hours.  The  goal 
was  to  minimize  time 
requirements. 

Objective  exceeded. 

•  The  control  identification  (pattern-matching) 
process  required  an  average  of  approximately  10 
minutes. 

•  Since  the  amount  of  problem-solving  required 
during  the  beta  test  was  intentionally  restricted  (to 
allow  statistical  comparisons),  it  is  expected  that 
the  actual  control  identification/summary  report 
process  will  require  an  average  of  approximately 

25  minutes. 
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Table  1.1 

Summary  of  Objectives  and  Performance  (Cont’d) 


Design 

Feature 

Air  Force 
Criteria/Objective 

Actual  Methodology  Performance 

Ease  of  control 
identification 

Enable  the  user  to  choose  from  a 
menu  of  control  options  (both 
short-term  and  long-term)  which 
when  implemented,  will 
minimize  the  risk  of 
musculoskeletal  disorders  and 
the  hazards  identified  within  the 
job  and  tasks. 

Objective  exceeded. 

•  The  Methodology  Guide  (Appendix  4)  includes 
comprehensive  Case  Study  Problem-Solving 
Matrices  which  provide  a  list  of  controls  for  50  of 
the  most  common  types  of 
maintenance/inspection  tasks. 

•  The  controls  are  identified  as 
modifications/adjustments  (e.g.,  short-term)  and 
major  changes  (long-term) 

•  Each  control  is  evaluated  for  its  expected  impact 
on  health/safety,  quality,  and  productivity. 

Ease  of  follow-up 
implementation 

Determine  which  type(s)  of 
additional  (Level  II)  analyses 
may  be  used  if  additional  data  is 
required. 

Objective  met. 

•  The  title  page  for  each  Case  Study  provides 

direction  on  the  type  of  Level  II  analysis  that  may 
be  completed  to  provide  greater  detail  on 
ergonomics  risk  factor  exposure. 

Ease  of  follow-up 
implementation 

The  Methodology  identifies 
metrics  which  will  be  used  to 
judge  the  impact  of  ergonomics 
improvements  on  employee 
health,  safety,  and  performance 
(e.g.,  quality,  productivity). 

Objective  met. 

•  The  title  page  for  each  Case  Study  provides 
recommendations  for  the  type  of  performance 
measures  that  may  be  used  to  justify  solutions 
and/or  measure  the  impact  of  those 
improvements. 

•  The  design  of  the  Level  I  Ergonomics 

Assessment  Checklist  provides  the  means  for  re¬ 
evaluating  the  job  after  changes  have  been  made. 
“Before”  and  “After”  Job,  Task,  and  Body 

Region  Priority  ratings  can  be  compared  to 
determine  the  immediate  benefits  to  employee 
health  and  safety. 
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DEVELOPMENT  PROCESS  SUMMARY 


2.1  USE  OF  THE  METHODOLOGY  IN  THE  TIERED  APPROACH  TO 

PROBLEM-SOLVING 

The  primary  goal  of  an  installation  ergonomics  program  is  to  prevent  work-related 
musculoskeletal  disorders  (WMD)  among  employees  routinely  exposed  to  ergonomics 
risk  factors.  The  basic  elements  of  an  installation  ergonomics  program  include:  Potential 
Ergonomics  Problem  Area  (PEP A)  designation,  Ergonomics  Problem  Area  (EPRA) 
designation  and  removal,  work  area  analysis,  medical  management,  and  training  and 
education.  Both  qualitative  (PEP A)  and  quantitative  (EPRA)  screening  techniques  are 
used  in  sequential  fashion  to  identify  employees  at  risk.  The  flowchart  in  Figure  2.1 
describes  the  ergonomics  program  process. 

Figure  2.1 

Ergonomics  Problem-Solving  Process 
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The  Level  I  Ergonomics  Assessment  and  Problem-Solving  Methodology  Guide  for 
Maintenance/Inspection  Work  Areas,  which  was  developed  on  the  basis  of  the  results 
discussed  in  this  Research  Report,  provides  BEF  with  a  process  for  conducting  a  basic 
ergonomics  assessment.  Through  use  of  a  simple  pattern-matching  process,  the  guide 
identifies  realistic  controls  that  will  effectively  minimize  or  eliminate  employee  exposure 
to  ergonomics  hazards  in  jobs  in  EPRA  designated  maintenance/inspection  areas. 


2.2  GENERAL  RATIONALE  FOR  THE  METHODOLOGY 

The  requirements  for  the  Methodology  design  were  specified  by  Headquarters  Air  Force 
Materiel  Command,  Office  of  the  Command  Surgeon  (HQ  AFMC/SGC)  and  Armstrong 
Laboratory.  Both  organizations  desired  an  effective  and  efficient  analysis  and  problem¬ 
solving  process  that  could  be  applied  to  the  full  variety  of  Air  Force  Maintenance  & 
Inspection  work  areas.  The  Methodology  was  to  be  designed  to  reflect  the  technical 
capabilities  of  a  Bioenvironmental  Engineering  (BEF)  technician  with  only  2-3  years  of 
experience.  In  addition,  the  process  was  to  place  primary  focus  on  identifying 
appropriate  controls.  Due  to  the  high  demands  already  placed  on  BEF  personnel  (e.g., 
responsibilities  for  non-ergonomics-related  activities)  and  the  potential  lack  of 
ergonomics  expertise,  the  Air  Force  requested  that  a  “pattern-matching”  process  be 
created  which  would: 

•  TTiinimiye  the  time  requirements  for  assessment  and  control  identification;  and 

•  enable  the  Air  Force  to  benefit  from  the  expertise  of  ergonomists  who  have 
had  years  of  experience  in  addressing  ergonomic  hazards  in 
maintenance/inspection  areas. 


2.2.1  Objectives.  The  Methodology  is  designed  to  enable  the  user,  primarily 

through  visual  observations  and  employ ee/supervisor  interviews,  to: 

•  identify  potentially  hazardous  tasks  within  a  shop  and  job; 

•  determine  if  the  content  of  the  job  and  task(s)  meet  established  ergonomics 
(risk  factor  exposure)  criteria; 

•  determine  which  type(s)  of  additional  (Level  II)  analyses  may  be  used  if 
further  quantification  of  ergonomics  hazards  is  required;  and 

•  choose  from  a  menu  of  control  options  (both  short-  and  long-term)  which, 
when  implemented,  minimize  the  risk  of  musculoskeletal  disorders  and  the 
hazards  identified  within  the  job  and  tasks. 
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The  Methodology  is  designed  to  enable  the  user  to  complete  data  collection  and  analysis 
on  a  maintenance/inspection  job  in  1-2  hours,  and  complete  the  control  identification  and 
summary  report  in  1-2  hours.  The  Methodology  includes  case  studies  for  typical 
maintenance/inspection  tasks.  The  case  studies  serve  as  the  basis  for  the  pattern- 
matching  process  that  will  be  used  to  “match”  the  hazards  identified  in  the  tasks  with 
controls  that  will  reduce  employee  exposure  to  those  hazards.  The  Methodology 
identifies  metrics  which  will  be  used  to  evaluate  the  impact  of  ergonomics  improvements 
on  employee  health,  safety,  and  performance  (e.g.,  quality,  productivity). 

2.2.2  Description.  Each  of  the  objectives  was  accomplished  by  developing  a 

Level  I  Ergonomics  Assessment  Checklist,  which  is  an  observation-based  “checklist”, 
and  Case  Study  Problem-Solving  Matrices,  which  provide  a  “controls  database”  for 
addressing  ergonomics  hazards.  The  Level  I  Ergonomics  Assessment  Checklist  requires 
no  measurement.  The  technician  collects  risk  factor  exposure  data  by  observing  the  task 
and  talking  with  the  employee.  The  checklist  results  can  be  directly  “matched”  to  the 
“database”  of  ergonomics  hazards  and  recommended  controls  provided  in  one  (or  more) 
of  50  Case  Study  Problem-Solving  Matrices.  The  matrices  are  designed  to  represent  the 
most  common  Air  Force  Maintenance/Inspection  tasks.  A  detailed  description  of  the 
pattern-matching  process  is  foimd  in  the  Level  I  Ergonomics  Assessment  and  Problem- 
Solving  Methodology  Guide  for  Maintenance/Inspection  Work  Areas. 


2.3  SCIENTIFIC  BASIS  FOR  THE  METHODOLOGY  DESIGN 

23.1  Literature  Review. 

2.3.1.1  Method.  A  literature  review  was  conducted  to  identify  existing 
methodologies  that  could  be  used  as  the  basis  for  the  Level  I  Methodology  for 
Maintenance/Inspection  Work  Areas.  Initial  results  of  the  review  indicated  that  at  the 
present  time,  comprehensive  ergonomics  analysis/problem-solving  methodologies,  which 
use  pattern-matching  as  the  basis  for  control  identification,  do  not  exist. 

As  a  result,  the  literature  review  was  targeted  to  identify  analysis  methods  upon  which 
one  part  of  the  Methodology,  the  Level  I  Ergonomics  Assessment  Checklist,  could  be 
based.  Sources  for  these  methods  included:  peer-reviewed  research  articles,  proposed 
volimtary  or  regulatory  standards,  and  proprietary  and  confidential  sources  of  the  analysis 
methods  that  were  identified  and  evaluated.  Three  were  drawn  from  standards,  eight 
were  drawn  fi-om  journals,  and  the  remaining  come  fi'om  proprietary  and  confidential 
sources.  We  have  included  a  list  of  the  proprietary  sources  that  we  could  use  without 
violating  any  copyrights  or  confidentiality  agreements. 
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An  evaluation  questionnaire  was  developed.  The  questionnaire  is  divided  into  three 
sections:  Subjective  Questions,  Objective  Questions,  and  Validation  Questions. 

2.3.1.2  Evaluation  Results.  Table  2. 1  summarizes  the  evaluation  results. 
Included  in  the  table  are  the  methods  which  achieved  the  highest  ratings  for  each  of  the 
objective,  subjective,  and  validation  questions.  The  feature(s)  of  the  method  which  was 
most  responsible  for  the  high  rating  is  also  discussed.  These  features  were  considered  as 
a  foundation  from  which  an  effective  Level  I  Ergonomics  Assessment  Checklist  could  be 
constructed.  In  some  instances,  the  evaluators  have  provided  comments  on  desirable 
features  even  though  none  of  the  methods  achieved  a  high  rating. 

2.3.1.3  Design  Objectives  for  the  Level  I  Ergonomics  Assessment.  As  a  result 
of  the  literature  review,  several  additional  design  objectives  were  identified  as  the  basis 
for  developing  the  Level  I  Ergonomics  Assessment.  These  objectives,  as  well  as  those 
specified  by  the  Air  Force,  are  listed  in  Table  2.2. 

The  remainder  of  this  Research  Report  provides  specific  information  on  the  practical  and 
scientific  basis  for  the  Guide  design. 
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Table  2.1 

Literature  Review  Results 


Factor 

Methods  Which  Achieved 
the  Highest  Ratings 

Desirable  Feature(s) 

1.  Face  Validity 

Checklist  seems  to 
measure  what  it  is 
supposed  to  measure 

•  OSHA  Draft  Checklist,  1995 
(U.S.  Dept,  of  Labor,  1995)[1] 

•  ANSI  Z365  Checklist,  1 995 
(Samples#!  and #2) (ANSI, 
1992)[2] 

•  List  of  risk  factors  compared  to  a  time 
exposure  scale 

•  All  major  risk  factors  considered 

2.  Overall 
UndgrstandabiUly 

When  you  first  look  at 
the  tool,  you  can 
immediately  tell  what  is 
going  on 

•  OSHA  Draft  Checklist,  1995  [1] 

•  ANSI  Z365  Checklist  [2] 
(Samples  #1  and  #2) 

•  PLIBEL,  Kemmlert  ( 1 994)[3] 

•  Job  Evaluation  Checklist,  1990 
(The  Joyce  Institute,  I990)[4] 

•  Matrix/table  format 

•  Risk  factors  listed  in  a  column 

•  Number  of  response  choices  limited  to 
reduce  variability 

1 

•  OSHA  Draft  Checklist,  1995  [1] 

•  Illustrations  provided  with  each 
question 

•  Concise  question  descriptions 

•  Exposure  cut-off  levels  clearly 
defined/quantified 

Questions 

Questions  are 
understandable  and  quick 
to  answer 

4,  Ease  of  Scoring 

Scoring  is 

understandable  and  quick 

•  ANSIZ-365(Sample#2)[2] 

•  Easiest  scoring  strategies:  addition  of 
a  column  of  numbers,  count 
occurrences  of  check  marks,  or 
picking  the  highest  number 

5.  Checklist  Completion 
Speed 

Can  be  filled  out  quickly 

•  ANSI  Z-365  (Sample  #2)  [2] 

•  Minimum  number  of  pages 

•  Minimum  number  of  questions 

•  Simple  scoring  procedures 

■  ■  nil  lillllllll  — 

•  ANSI  Z-365  (Sample  #2)  [2] 

•  Single  page  format 

•  Large  size  text  fonts 

•  Reasonable  levels  of  information  on  a 
page 

Work  site 

Can  be  easily  completed 
at  the  work  site 

7.  Objectivity 

•  ANSI  Z-365  (Sample  #1)  [2] 

•  OSHA  Draft  Checklist  [1] 

•  Well-defined  questions 

•  Exposure  levels  clearly 
defined/quantified 

•  Scoring  levels  quantified  and 
operationally  defined 

The  questions  and 
scoring  are  not  impacted 
by  different 
interpretations 
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Table  2.1 

Literature  Review  Results  (Cont’d) 


Factor 

Methods  Which  Achieved 
the  Highest  Ratings 

Desirable  Feature(s) 

•  OSHA  Draft  Checklist  [1] 

•  ANSI  Z-365  (Sample  #2)  [2] 

•  No  technical  jargon  (uses  “reaching” 

instead  of  “shoulder  abducted”) 

9.  Relevance  of 

Questions  to  Typical 

Work  Situations 

Questions  reflect  realities 
in  the  work  environment 

•  OSHA  Draft  Checklist  [1] 

•  Office/Computer  Checklist  [5] 

•  Illustrations  of  “real”  work  situations 

•  Questions  worded  to  provide 
examples  of  “real  world”  occurrences 
of  risk  factors 

10.  Flexibility 

Checklist  usable  for  a 
variety  of  different  types 
of  tasks 

•  Uniroyal  Industrial  Ergonomics 
Checklist  (The  Joyce  Institute, 
1995)[6] 

•  OSHA  Draft  Checklist  [1] 

•  Questions  address  risk  factors  which 
can  occur  in  a  wide  variety  of  tasks 

•  Address  visual,  environmental  issues 

•  Job  Evaluation  Checklist  [4] 

•  OSHA  Draft  Checklist  [1] 

•  Ratings  provided  for  multiple  body 
regions  and  the  overall  job 

•  Provides  direction  for  problem  solving 

12.  Equipment  Required 

•  ANSI  Z-365  (Sample  #2)  [2] 

•  Uniroyal  Industrial  Ergonomics 
Checklist  [6] 

•  No  equipment  (e.g.,  calculator  or 
computer)  should  be  required 

•  ANSI  Z-365  (Samples  #1  &  2) 

[2] 

•  Uniroyal  Industrial  Ergonomics 
Checklist  [6] 

•  Design  checklist,  Lifshitz  & 
Armstrong,  1986  [7] 

•  Minimal  number  of  pages  desirable 

•  PLIBEL  [3] 

•  RULA,  McAtamney  &  Corlett, 
1993  [8] 

•  Upper  Extremity  Checklist, 
Keyserling  et.  al.,  1993  [9] 

•  Design  checklist  [7] 

•  Ergonomics  job  analysis 
method.  Stetson  et.  al.,  1991 
[10] 

•  Tool  supported  by  research 

•  Ergonomicsjob  analysis  method 
[10] 

•  Research  conducted,  favorable  result 

•  Clearly  defined  questions  and  rating 
scales 

•  Minimum  interpretation  required 

•  Resistant  to  minor  changes  in  worker 
technique 
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Table  2.1 

Literature  Review  Results  ( 


Factor 

Methods  Which  Achieved 
the  Highest  Ratings 

16.  Inter-Anaivst 

Reliability 

Multiple  analysts 
complete  the 
checklist/reliability 
demonstrated 

•  RULA  [8] 

•  PLIBEL  [3] 

•  Ergonomics  job  analysis  method 
[10] 

17.  Method  Sensitivity 
Discriminates  between 
low  and  high  risk  jobs 

•  RULA  [8] 

18.  Content  Validity 

Uses  measures  supported 
in  literature 

•  PLIBEL  [3] 

•  PLIBEL  [3] 

•  Upper  Extremity  Checklist  [9] 

Provides  information 
close  in  value  to  that 
from  a  more  detailed 
analysis 

1 

•  Design  checklist  [7] 

•  RULA  [8] 

Results  predict  or 
correlate  with  injuries, 
complaints,  symptoms 

•  Upper  Extremity  Checklist  [9] 

Use 

Measures  of  overall  ease 
of  use  and  applicability 

•  Ergonomics  job  analysis  method 
[10] 

Minimal  time  required  to 
complete  checklist 

lA 


:’d) 


Desirable  Feature(s) 


•  Research  conducted,  favorable  result 

•  Clearly  defined  questions  and  rating 
scales 

•  Minimum  interpretation  required 

•  Resistant  to  minor  changes  in  worker 

technique _ 

•  Research  conducted,  favorable  result 

•  Time  exposure  scale  allows  varying 
levels  of  exposure 

•  Minimum  number  of  questions _ 

•  Research  conducted,  favorable  result 

•  Questions/Job  Factors  are  supported  in 

the  literature. _ 

•  Research  conducted,  favorable  result 

•  Collects  similar  information  to  that 

obtained  by  more  detailed  analysis  but 
collects  the  information  in  a  more 
concise  manner _ 

•  Research  conducted,  favorable  result 

•  Job  factor  questions  supported  by 

research  which  indicates  a  relationship 
between  the  job  factor  and  discomfort 
or  injuries. _ 

•  Research  conducted,  favorable  result 

•  Matrix/table  format 

•  Illustrations  for  each  question 

•  Well-defined  questions 

•  Simple  scoring  procedures _ 

•  Research  conducted,  favorable  result 

•  Minimum  number  of  questions 

•  Minimum  number  of  pages 

•  Completion  time  meets  or  is  under 
specified  completion  time  objectives 


Table  2.2 

Development  Objectives  for  the  Level  I  Ergonomics  Assessment 


Air  Force  Objectives 


Task-based,  problem-solving  checklist _ 

Complete  by  BEF  technician  without  assistance  from  engineer _ 

Include  visual,  musculoskeletal,  environmental  issues  (organizational  issues 

excluded  by  request  of  Air  Force) _ _ 

Checklist  completion  based  on  real-time  observation  as  well  as  employee  interview 

Level  of  musculoskeletal  stress  rated  by  body  part  and  overall  for  the  job _ 

Provides  numerical  “priority”  rating  scale  _ 

Analysis  of  results  indicates  which  aspects  of  the  task  and/or  workplace  may  be 

modified  to  achieve  the  most  significant  improvement _ _ 

Indicates  the  type  of  detailed  analysis  (Level  II)  which  is  most  appropriate  when 

quantification  of  risk  factors  is  desired _ _ _ 

Enables  technician  to  measure  results/impact  of  workplace  improvements _ 

Applicable  to  majority  of  maintenance/inspection  jobs _ 

Consistent  with  Job  Requirements  and  Physical  Demands  Survey  (administered  by 
PHF) _ 


Additional  Objectives  Based  on  Literature  Review 


Considers  all  risk  factors  supported  by  the  literature  _ 

Avoids  the  need  for  taking  physical  measurements  _ 

Provides  examples  and  illustrations  of  real  world  occurrences  of  risk  factors  for  each 

question _ _ _ _ _ 

Questions  designed  such  that  the  assessment  is  resistant  to  minor  changes  in  worker 

technique _ _ _ _ _ 

MinimiTes  the  use  of  technical  “jargon”  _ 

Provides  clear  and  concise  definition  for  questions  and  rating  scales;  minimize  the 

need  for  interpretation  _ _ _ _ _ 

Uses  a  time-exposure  scale  for  making  judgments  about  risk  factor;  include  exposure 

fi'equency  and  task  duration _ _ _ 

Defines/quantifies  exposure  levels _ 

Minimizes  calculations  required  for  scoring  _ 

Provides  risk/priority  ratings  for  multiple  body  regions  and  overall  job _ 

Provides  numerical  results  and  operationally  defines  scores _ _ 

Requires  a  minimum  number  of  pages  _ 

Uses  large  text/fonts _ _ _ 


2-8 


3.0 


PRACTICAL  AND  SCIENTIFIC  BASIS  FOR  LEVEL  I 
ERGONOMICS  ASSESSMENT  CHECKLIST 


The  Level  I  Ergonomics  Assessment  Checklist  is  comprised  of  four  parts: 

•  Part  I  -  Work  Content 

•  Part  II  -  Job  Factors 

-  Shoulder/Neck 

-  HandsAVrists/Arms 

-  Back/Torso 
—  Legs/Feet 

-  Head/Eyes 

•  Part  III  -  Environmental  Factors 

•  Part  IV  -  Employee  Suggestions 

3.1  PARTI:  WORK  CONTENT 

The  list  of  tasks  included  in  the  Task  Key  List  were  taken  from  Part  III  of  the  Job 
Requirements  and  Physical  Demands  Survey  (administered  by  PHF).  These  tasks  were 
extracted  from  the  Dictionary  of  Occupational  Titles  (U.S.  Dept,  of  Labor,  1991).[1 1] 

Additional  tasks  were  added  to  improve  the  breadth  of  the  task  list.  These  additional 
tasks  are  presented  in  Table  3.1. 

Table  3.1 

Tasks  Added  to  the  Work  Content  Matrix 


Task  List 

121. 

assembly/disassembly  internal  component 

122. 

assembly/repair  bench  work 

123. 

computer  work 

124. 

hose  handling 

125. 

forming 

126. 

masking 

127. 

media  blasting  (blast  cabinet) 

128, 

media  blasting  (high  pressure  gun) 

129. 

ordnance  disposal 

130, 

prying 

131. 

visual  inspection 
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Critical  tasks  are  the  tasks  considered  during  checklist  completion.  The  cut-off  for 
critical  tasks  used  in  the  Administrative  Guide  was  one  or  more  hours  (See  Research 
Report  for  Administrative  Work  Areas).  Many  Maintenance  and  Inspection  (M/I)  tasks, 
however,  are  not  performed  on  a  daily  schedule.  For  example,  a  typical  M/I  job/task 
might  be  performed  one  day  a  week  or  three  weeks  out  of  the  year.  Due  to  the  increased 
variability  of  job/task  performance  ffequency/duration,  it  was  determined  that  a 
percentage  scale  expressed  as  the,  “total  percentage  of  the  work  time  spent  performing 
the  task”,  would  be  a  more  appropriate  Task  Frequency  scale. 

The  cut-off  for  critical  tasks  of  10%  of  work  time  was  selected  to  be  consistent  with  the 
Administrative  Guide  (1  hr  /  8  hr.  =  12.5%)  but  also  simple  and  easy  to  remember  (12.5% 
was  rounded  down  to  10%).  The  cut-off  for  the  distinction  between  moderate  and  high 
frequency  of  50%  of  work  time  was  also  derived  from  the  Administrative  Guide  (4  hr.  /8 
hr.  =  50%). 

The  decision  about  Critical  Tasks  suggests  that  when  priorities  are  established  for 
problem-solving,  tasks  which  occur  for  less  than  10%  of  work  time  (total  accumulated 
time  performing  the  task)  should  not  be  considered  as  a  primary  concern.  This  is 
consistent  with  the  hazard-control  prioritizing  philosophy  included  in  the  AFOSH 
program. 

The  exception  to  this  rule  is  for  tasks  which  involve  lifting  or  high  forces.  These  tasks 
are  to  be  considered  critical  tasks  regardless  of  the  frequency. 


3.2  PART  II:  JOB  FACTORS 

3.2.1  Format.  The  checklist  design  and  question/response  format  was  selected 

to  achieve  maximum  consistency  with  the  JR/PD  Survey  while  still  providing  a  tool 
which  is  sensitive  enough  to  quantify  risk  factors  at  the  job/task  level. 

Part  II  is  divided  into  five  sections  by  body  region.  These  body  regions  correspond  to  the 
body  region  structure  employed  in  the  survey  tool  and  the  individual  risk  factors  are 
associated  with  each  body  region.  This  structure  is  similar  to  that  used  in  the  PLIBEL 
analysis  method  [3]. 

The  checklist  is  designed  as  a  table  or  matrix  which  lists  Job  Factors  (risk  factors)  in  the 
left  column  and  frequency/exposure-based  responses  to  the  right.  This  table  format  is 
widely  used  for  “Level  I”  analysis  methods  (e.g.,  OSHA  Draft  Ergonomics  Checklist[l]; 
ANSI  Z-365  Draft  Checklists  [2],  Upper  Extremity  Checklist  [9],  Ergonomic  Job 
Analysis  Method  [10])  because  it  is  fast  and  easy  to  complete. 
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To  satisfy  the  objective  for  providing  a  task-based  problem-solving  Methodology,  the 
Level  I  Assessment  Checklist  format  enables  the  user  to  assess  exposure  to  ergonomics 
risk  factors  for  each  of  the  tasks  that  make  up  the  larger  job.  This  feature  is  unique  to  the 
Level  I  Ergonomics  Assessment  Checklist  and  provides  the  basis  for  assessing  the  overall 
job  exposure  (for  within-shop  Prioritization  purposes),  while  simultaneously  providing 
individual  task  exposure  information.  The  task  exposure  data  will  enable  the  technician 
to  prioritize  the  focus  of  problem-solving  efforts  based  on  the  individual  task  which 
contributes  most  significantly  to  the  overall  risk  factor  exposure. 

3.2.2  Question  Selection.  Job  Factor  questions  were  based  on  specific 
ergonomics  risk  factors  reported  to  contribute  to  WMDs.  When  appropriate,  questions 
from  the  JR/PD  Survey  and  Administrative  Guide  were  incorporated  into  this  checklist  to 
maintain  consistency.  Additional  Job  Factor  questions  were  drawn  from  or  based  on 
analysis  methods  reported  in  peer  review  journals,  from  existing  volrmtary  standards 
(e.g.,  OSHA  Draft  Ergonomics  Standard  [1],  ANSI  Z-365  Draft  Standard  [2],  ANSI-HFS 
100-1988  VDT  Standard  [12]),  or  extrapolated  from  established  risk  factors  (e.g., 
stressful  postures,  excessive  force,  etc.). 

The  basis  for  each  of  the  Job  Factor  questions  is  presented  in  Table  3.2.  The  table  notes 
if  reliability  and/or  validity  has  been  reported  for  each  question  and  specifies  if  the 
question  was  substantially  altered  for  inclusion  in  the  Level  I  Ergonomics  Assessment 
Checklist.  (Note:  Minor  modifications  were  made  to  all  questions  in  order  to  maximize 
consistency  in  verbiage  and  presentation.) 

Questions  were  combined  whenever  possible  to  improve  scoring  accuracy.  If  the  co¬ 
occurrence  of  two  Job  Factors  do  not  produce  an  additive  effect  on  overall  priority  scores, 
then  the  Job  Factors  were  combined  into  one  question.  For  instance,  since  carrying  of 
materials  and  repetitive  arm  forces  both  produce  forces  in  the  arm  and  shoulder,  these  two 
Job  Factors  were  combined  into  one  question. 

3.2.3  Question/Response  Structure  and  Scientific  Basis.  The  checklist  is 
composed  of  a  series  of  questions  which  describe  postural  deviations,  forces,  or  localized 
contact  stresses  for  each  region  of  the  body.  Each  question  asks  if  the  Job  Factor  exists  in 
the  task  being  evaluated.  If  the  Job  Factor  is  identified,  the  checklist  asks  for  a  response 
which  estimates  the  time-based  exposure  to  that  risk  factor.  The  checklist  design 
automatically  assigns  a  score  which  corresponds  to  the  estimated  exposure  time  for  that 
Job  Factor. 
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Table  3.2 

Original  Source  and  Treatment  of  Job  Factor  Questions 


Question 

Number 

Original  Source 

Adapted 

from 

RF/DS 

Reliability 
&  Validity 
Reported 

Substantial 

Alterations 

from 

Original 

1 

RULA  [8]  p91-99 

Chaffin  and  Andersson  (1984)  [13]  pp 
336-341 

PLffiEL  [3]  pl-21 

Keyserling  et.  al.  (1993)  [9]  p81 1 

Kiibom  &  Persson  (1987)  [14]  p276 
Putz-Anderson  (1992)  [15]  p52 

Punnett  et.  al.  (1985)  [16]  p417-425 
Sommerich  et.  al.  (1993)  [17]  p697-717 
Hagberg  (1984)  [18]  p269-278 

Yes 

RF/DS  #5 

Yes 

Yes 

2 

Stetson  et.  al  1991)  [10]  p935 

Van  Cott  &  Kincade  (1972)  [19]  p72 
Putz-Anderson  (1992)  [15]  p62 

Snook  &  Ciriello  [20]  pl210 

Wells  et.  al.  (1983)  [21]  p814-820 

Eastman  Kodak  (1986)  [22]  Vol.2,  pl41 

Yes 

RF/DS  #3/#4 

Yes 

Yes 

3 

Schoenmarklin,  Marias  &  Leurgans  (1994) 
[23]  pl449-1459 

Yes 

RF/DS  #4 

No 

Yes 

4 

Chaffm  &  Andersson  (1988)  [24]  p  347 
Chaffin  (1973)  [25]  p346-354 

Kiibom,  Persson  &  Jonsson  (1986)  [26] 
p37-47 

RULA  [8]  p91-99 

ANSI  Z-365  [2] 

OSHA  draft  checklist  [1] 

Kiibom  &  Persson  [14]  p275 

Yes 

RF/DS  #6/#7 

Yes 

Yes 

5 

Putz-Anderson  (1992)  [27]  p23 

Keyserling  et.  al.  (1993)  [9]  p81 1 

Stetson  et.  al  1991)  [10]  p927-937 
Silverstein  et.  al.  (1987)  [28]  p343-358 
Hammer  (1934)  pl39-140 

OSHA  draft  checklist  [1] 

Yes 

RF/DS 

#8/#9/#18 

Yes 

Yes 

6 

Hammer  (1934)  [29]  pl39-140 

Silverstein  et.  al.  (1987)  [27] 

HUH 

No 

Yes 

7 

ANSI  Z-365[2] 

Putz-Anderson  (1992)  [30]  p66 

Yes 

RF/DS  #20 

No 

Yes 
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Table  3.2 

Original  Source  and  Treatment  of  Job  Factor  Questions  (Cont’d) 


Question 

Number 

Original  Source 

Adapted 

from 

RF/DS 

Reliability 
&  Validity 
Reported 

Substantial 

Alterations 

from 

Original 

8 

Keyserling  et.  al.  (1993)  [9]  p810 

Van  Cott  &  Kincade  (1972)  [19] 

Stetson  et.  al.  (1991)  [10]  pp  935-936 
Putz-Anderson  (1992)  [15]  p62 

Yes 

RF/DS  #21 

Yes 

Yes 

9 

Schoenmarklin  et.  al.  (1994)[23] 
Putz-Anderson  (1992)  [15]  p68 

Keyserling  et.  al.  (1993)[9]  p81 1 

Yes 

RF/DS 

#13/#16/#17 

Yes 

Yes 

10 

Putz-Anderson  (1992)  [15]  p68-69 
Keyserling  et  al.  (1993)  [9]  p81 1 

OSHA  draft  checklist  [1] 

Yes 

RF/DS  #12 

Yes 

Yes 

11 

Keyserling  et  al.  (1993)  [9]  p81 1 

No 

Yes 

Yes 

12 

McAtamney  and  Corlett  (1993)  [8]  p93-94 
Keyserling  (1986)  [31]  p^l-649 

OSHA  draft  checklist  [1] 

Kemmlert  (1994)  PLIBEL  [3] 

Yes 

RF/DS  #25 

Yes 

Yes 

13 

Marras  et  al.  (1993)  [32]  pl55-158 

Yes 

RF/DS 

#25/#26 

No 

Yes 

14 

Marras  et  al.  (1993)  [32]  pl55-158 

Yes 

RF/DS  #26 

No 

Yes 

15 

OSHA  draft  checklist  [1] 

RULA  [8]  p94 

Keyserling  (1986)  [31] 

Yes 

RF/DS  #23 

Yes 

Yes 

16 

NIOSH  Lifting  Guide  [33] 

OSHA  draft  checklist  (manual  handling 
portion)  [1] 

Yes 

RF/DS  #30 

Yes 

Yes 

17 

Snook  &  Ciriello  [20]  pi  197-1213 

No 

Yes 

No 

18 

NIOSH  Vibration  Standard  [34]  p26-48 

Yes 

RF/DS  #27 

Yes 

Yes 

19 

OSHA  draft  checklist  [1] 

Kemmlert  (1994)  PLIBEL  [3]  pl-21 

Yes 

RF/DS  #34 

Yes 

Yes 

20 

OSHA  draft  checklist  [1] 

No 

No 

Yes 

21 

Manilla  et.  al.  (i993)  [35]  p405-41 1 

OSHA  draft  checklist  [1] 

Yes 

RF/DS  #31 

Yes 

No 

22 

OSHA  draft  checklist  [1] 

Yes 

RF/DS  #32 

No 

Yes 

23 

ANSIZ-365[2]pll 

Yes 

RF/DS  #35/ 

#38 

No 

Yes 

24 

Bergqvist  (1995)  [36]  p201-209 

Yes 

RF/DS  #37 

No 

Yes 
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A  Isrge  body  of  research  confirms  that  the  existence  of  postural  deviations,  forces, 
localized  contact  stress,  and  high  repetition  rates  (or  continuous  exposure)  in  a  job  can 
lead  to  muscular  fatigue  and  WMDs.  These  factors  are  called,  primary  risk  factors.  A 
complete  listing  of  references  for  each  of  the  primary  risk  factors  is  provided  in 
Keyserling  [9, 37]. 

Unfortimately,  however,  the  precise  magnitude/duration  of  exposure  to  each  Job  Factor 
that  causes  WMDs  has  not  been  determined  through  research.  This  research  report  does 
not  attempt  to  establish  a  cause-effect  relationship.  Still,  some  basic  concepts  are 
understood  regarding  how  these  risk/job  may  factors  interact  and  contribute  to  localized 
discomfort,  pain,  etc.,  which  are  recognized  as  pre-cursors  to  WMDs.  Silverstein,  et.  al. 
[38,39]  foimd  that  repetition  (i.e.,  exposure  time)  and  force  have  a  multiplicative  effect 
when  they  co-occur  in  a  job.  Reynolds,  et.  al.  [40]  added  postural  deviation  to  this 
equation,  suggesting  that  exposure  can  be  defined  as  the  umon  of  force  and  postural 
deviation  multiplied  by  the  daily  frequency  (i.e.  amount  of  exposure).  Postures  can  be 
thought  of  as  forces  exerted  internally  in  the  body.  Localized  contact  stress  can  also  be 
thought  of  as  a  force  exerted  by  an  external  object.  All  risk  factors  can  be  translated  into 
a  force  experienced  internally  within  some  region  of  the  body  over  a  period  of  time. 

Thus,  an  approach  which  compares  a  series  of  posture  and  force-related  Job  Factor 
questions  to  an  exposure  scale,  has  been  suggested  in  the  literature. 

For  the  Level  I  Ergonomics  Assessment  Checklist,  exposure  time  is  measured  by 
considering  the  total  task  duration  and  as  well  as  proportion  of  the  task  time  in 
which  the  Job  Factor  occurs.  Both  task  duration  and  prevalence  of  the  Job  Factor  within 
the  task  are  critical  to  making  a  correct  assessment  of  the  total  exposure  to  that  risk 
factor.  For  example,  a  task  may  be  performed  all  day  long  (e.g.,  7.5  hours  per  day)  but 
the  prevalence  of  the  Job  Factor  is  low  (e.g.,  5%  of  the  task).  Conversely,  the  prevalence 
of  the  Job  Factor  may  be  high  (e.g.,  performed  continuously,  40  times/minute)  but  the 
task  duration  may  be  low  (e.g.,  30  minutes  per  day).  The  highest  exposure  of  a  Job 
Factor  occurs  when  both  the  task  duration  and  the  Job  Factor  prevalence  within  that  task 
are  high.  It  is  critical  to  be  able  to  discriminate  between  different  levels  of  task  duration 
and  Job  Factor  prevalence  in  order  to  accurately  assess  and  prioritize  jobs  for  corrective 
action  based  on  overall  exposure.  Notice,  as  well,  that  Job  Factor  prevalence  can  be 
defined  in  a  number  of  different  ways  (e.g.,  as  a  repetition  rate  or  as  a  percentage  of  time) 
depending  on  if  the  activity  is  continuous  or  repetitive. 

The  more  common  checklists  used  in  industry  seek  to  address  the  issue  of  task  duration 
and  Job  Factor  prevalence  in  a  number  of  different  ways.  Unfortunately,  only  a  small 
number  of  these  checklist  tools  have  been  formally  validated.  Furthermore,  there  is  even 
less  research  which  actually  prescribes  the  most  effective  way  to  consider  both  task 
duration  and  Job  Factor  prevalence. 
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The  sample  checklist  provided  in  the  OSHA  Draft  Standard  [1]  uses  a  pre-screening 
approach  for  which  the  user  first  considers  task  duration  then  considers  the  duration  of 
the  risk  factor  within  each  task.  Finally,  the  user  responds  to  each  risk  factor  question  by 
indicating  the  total  daily  duration  of  the  risk  factor  in  all  tasks.  This  approach  has  a 
significant  drawback  in  that  an  extra  step  is  required  to  estimate  risk  factor  duration.  In 
addition,  this  approach  is  more  effective  for  estimating  exposure  from  continuous 
postures  and  less  effective  for  repetitive  movements. 

The  ANSI  Z-365  Sample  Risk  Factor  Checklist  (Sample  #1)  [2]  takes  into  accoimt  both 
total  task  duration  and  Job  Factor  prevalence  (expressed  as  “repetition  rate”).  Reynolds, 
et.  al.  [40]  expresses  exposure  as  daily  output  (or  number  of  repetitions  per  day).  This 
works  well  for  repetitive  activities  but  is  not  as  effective  for  continuous  postures. 

3.2.4  Scoring. 

3.2.4.1  Scaling  Function.  Three  scoring  levels  were  selected  as  response  choices 

for  each  Job  Factor:  “Frequently”,  “Sometimes”,  “Occasionally”,  and  “Never”.  The 
definitions  of  each  exposure  level  are  defined  below: 


For  each  Job  Factor,  score: 

•  Frequently  (F):  Job  Factor  occurs  for  greater  than  50%  of  the  task 

•  Sometimes  (S):  Job  Factor  occurs  for  10-50%  of  the  task 

•  Occasionally  (O):  Job  Factor  occurs  for  less  than  1 0%  of  the  task 

•  Never  (N):  if  the  Job  Factor  does  not  occur  OR  does  not  apply. 


The  explicit  division  between  task  frequency  and  Job  Factor  prevalence  is  a  departure 
from  the  Administrative  Guide.  In  the  Maintenance  and  Inspection  Guide  there  is  one 
scale  to  assess  task  frequency  and  a  different  scale  to  assess  Job  Factor  prevalence.  This 
change  was  made  to  reduce  the  tendency  to  confuse  task  frequency  with  Job  Factor 
prevalence.  This  is  a  particular  concern  for  maintenance  and  inspection  tasks  because  of 
the  high  variability  in  both  task  frequency  and  Job  Factor  prevalence.  In  this  new 
approach,  decisions  of  how  frequently  the  task  is  performed  are  made  distinct  from 
decisions  of  how  prevalent  the  Job  Factor  is  in  the  task. 

The  “Occasionally”  category  (less  than  10%  of  the  task)  was  added  to  increase  the 
sensitivity  of  the  tool  in  capturing  high  force  activities  which  may  take  a  relatively  small 
proportion  of  the  task  time,  but  which  could  have  a  significant  impact  on  the  risk  of 
injury.  The  “Occasionally”  category  also  helps  the  tool  to  detect  a  cumulative  effect 
across  Job  Factors  and  across  tasks. 
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Obtaining  an  estimate  of  the  multiplicative  effect  between  force  and  exposure  time  is 
accomplished  through  the  use  of  a  higher  score  for  Frequently  responses,  and  a  lower 
score  for  Sometimes  and  Occasionally  responses.  The  use  of  a  Job  Factor  prevalence  cut¬ 
off  of  greater  than  50%  of  the  task  time  is  supported  by  Silverstein  et.  al.  [39]. 

Some  Job  Factor  questions  were  given  higher  or  lower  scores  than  other  Job  Factor 
questions  (e.g.,  F=3  S=2  0=1  vs.  F=6  S=3  0=1).  This  scoring  strategy  enables  the 
Checklist  to  provide  the  appropriate  “weight”  to  Job  Factors  which  have  a  lesser  or 
greater  impact  on  the  development  of  discomfort.  The  weighting  was  based  on  a 
systematic  process  on  comparing  all  possible  pairs  of  Job  Factor  scores  for  a  particular 
body  region.  The  objective  was  to  assess  whether  combined  scores  matched  the 
ergonomists’  expectations  of  the  musculoskeletal  risk  associated  with  that  pair  of  Job 
Factors.  This  process  was  also  conducted  for  individual  Job  Factors. 

The  scoring  for  Job  Factors  considers  the  impact  of  task  frequency  and  Job  Factor 
prevalence  for  determining  the  impact  of  the  Job  Factor  on  musculoskeletal  risk.  For 
example,  for  Question  #16.  Lifting  forces,  F=7,  S=6, 0=4,  N=0  was  used  to  indicate  that 
even  if  the  amount  of  time  exposed  to  the  Job  Factor  is  low,  the  musculoskeletal  risk  can 
still  be  significant. 

3.2.4.2  Body  Region  Scoring.  For  each  task,  within  a  single  body  region,  the 

scores  for  all  Job  Factors  identified  are  summed.  This  summative  operation  simulates  the 
summative  affect  across  risk  factors  on  a  single  body  region.  For  instance,  consider  a 
person  who  is  applying  high  forces  while  working  with  the  arms  held  away  from  the  body 
unsupported.  In  this  case,  one  would  expect  that  a  higher  priority  score  would  be 
obtained  than  if  the  person  were  applying  high  force  without  reaching. 

Task  scores  represent  the  column  total  for  all  Job  Factors  in  a  single  body  region.  These 
task  scores  are  transferred  to  the  Checklist  Scoring  Summary  (Section  2,  Scoring 
Summary). 

33  PARXni:  CHECKLIST/ENVIRONMENTAL  FACTORS 

33.1  Questions.  The  environmental  factors  are  designed  and  listed  in  a  format 

similar  to  that  used  for  the  body  regions.  The  basis  for  the  questions  is  presented  in  Table 


3-8 


Table  3.3 

Original  Source  and  Treatment  of  Environmental  Factor  Questions 


Question 

Number 

Original  Source 

Adapted 

from 

RF/DS 

Reliability 
&  Validity 
Reported 

Substantial 

Alterations 

from 

Original 

25 

Canadian  Standard  [41]  p.80 

No 

No 

Yes 

26 

USAF  Technical  Report  on  Indoor  Air 
Quality  [44]  p.24 

Canadian  Standard  [42]  p.69 

No 

No 

Yes 

27 

Canadian  Standard  [43]  p.63 

AFOSH  STD  48-19  [45]  ch.2 

Yes 

RF/DS  #36 

No 

Yes 

28 

USAF  Technical  Report  on  Indoor  Air 
Quality  [44]  p.24 

Canadian  Standard  [46]  p.  72-75 

No 

No 

Yes 

3.3.2  Scoring.  The  Environmental  Factors  are  scored  on  an  agreement  scale.  A 

score  of  4  is  given  for  “Strongly  Agree”  responses  while  a  score  of  1  is  given  for  “Agree” 
responses.  A  score  of  0  is  given  for  all  other  responses. 

The  environmental  rating  scale  was  established  to  correspond  to  the  priority  rating  scale 
used  for  the  body  regions.  This  scale  also  has  three  levels:  high,  medium,  and  low.  The 
logic  for  the  cut-offs  for  high,  medium,  and  low  are  based  on  a  consensus  judgment  of 
TJI/ADL  ergonomists: 

•  A  medium  priority  rating  corresponds  to  one  Strongly  Agree  environmental 
factor  or  four  Agree  environmental  factors. 

•  A  high  priority  rating  corresponds  to  two  Strongly  Agree  environmental 
factors  or  eight  Agree  environmental  factors. 

Ergonomist  consensus  considered  the  likelihood  that  employee  perceived  exposxire  to 
several  Environmental  Factors  would  cause  the  employee  to  report  a  concern.  The 
consensus  also  felt  that  environmental  factors  might  also  adversely  affect  job 
performance. 


3.4  PART  IV:  EMPLOYEE  SUGGESTIONS 

This  section  is  included  to  reflect  the  benefits  of  employee  involvement.  While 
employees  may  not  be  qualified  to  conduct  the  Level  I  Ergonomics  Assessment 
Checklist,  they  can  provide  helpful  suggestions  and  invaluable  assistance  for  prioritizing 
the  focus  of  any  problem-solving  efforts.  The  input  requested  on  the  Level  I  Checklist  is 
more  task  specific  than  that  which  was  requested  about  the  potential  for  improving  work 
in  general  throughout  the  shop. 
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FINAL  SCORING  AND  RESULTS  INTERPRETATION 


A  tabulation  matrix  (Scoring  Summary  section  of  the  Ergonomics  Summary  Report)  is 
used  to  calculate  the  priority  ratings  for  each  body  region,  each  task,  and  for  the  overall 
job. 

The  Scoring  Summary  design  resulted  from  a  combination  of  findings  from  the 
literature  review  as  well  as  the  consensus  judgment  from  experienced  TJI/ADL 
ergonomists.  In  the  literature,  there  is  a  lack  of  validated  methods  for  determining  a  “cut¬ 
off’  between  “ergonomics  problem/risk  of  WMD”  and  “no  ergonomic  problem/no  risk  of 
WMD.”  Therefore,  the  scoring  concept  and  results  generated  by  the  Checklist  are 
designed  to  prioritize  the  need  for  corrective  action  based  on  the  highest  exposure  to 
ergonomics  risk  factors.  In  other  words,  a  “High”  rating  means  that  exposure  to  risk 
factors  which  have  been  associated  with  WMDs  is  high.  It  does  not  mean  that  the  risk 
for  injury  is  high.  The  ergonomics  research  community  continues  to  work  towards 
providing  a  dose-response  relationship.  When  interpreting  results,  the  technician  is 
instructed  to  focus  problem-solving  efforts  on  any  job,  task,  body  region  which  is  rated 
“high”  or  “medium.” 

The  priority  rating  scale  has  three  levels:  high,  medium,  and  low.  The  logic  for  the  cut¬ 
offs  for  high,  medium,  and  low  are  based  on  the  following  rationale: 

•  a  medium  priority  rating  corresponds  to  exposure  to  one  Frequently  Job 
Factor  or  four  Sometimes  Job  Factors;  and 

•  a  high  priority  rating  corresponds  to  exposure  to  two  Frequently  Job  Factors 
or  eight  Sometimes  Job  Factors. 

Priority  scores  are  generated  for  each  Body  Region,  for  each  Task,  and  for  the  Overall 
Job. 


3.5.1  Body  Region  Score.  Body  Region  Scores  for  each  task  are  determined  by 

totaling  the  responses  to  the  Job  Factor  questions  for  each  task.  Body  Region  scores  for 
the  job  as  a  whole  are  determined  by  averaging  scores  across  tasks.  The  averaging 
process  was  selected  by  TJI/ADL  ergonomists  to  reflect  the  beneficial  impact  of  task 
variety.  Consider  the  following  example  jobs. 

Job  A  is  comprised  of  just  one  task:  welding.  This  task  exposes  the  shoulder/neck  to  a 
high  level  of  ergonomics  risk  factors  -  Body  Part  Score  =  8.  Since  there  is  only  one  task, 
the  Body  Region  Priority  Score  is  8,  which  is  a  “High”  rating. 

Job  B  is  comprised  of  two  tasks;  welding  and  computer  work.  This  welding  task,  which 
is  performed  for  5  hours  per  day,  also  exposes  the  shoulder/neck  to  a  “high”  level  of 
ergonomics  risk  factors  -  Body  Part  Score  =  8.  The  computer  work,  which  is  performed 
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for  3  hours  per  day,  exposes  the  shoulder/neck  to  a  “low”  level  of  ergonomics  risk  factors 
-  Body  Part  Score  =  2.  The  average  Body  Region  Priority  Score  is  5,  which  is  a 
“Medium”  rating. 

A  comparison  of  the  Body  Region  Priority  Score  for  each  task  suggests  that  Job  B  is 
easier  on  the  shoulder  than  Job  A.  The  “Medium”  rating  on  Job  B  suggests  that,  since  the 
employee  spends  part  of  the  day  performing  a  task  (computer  work),  which  provides 
some  relief  to  the  shoulder,  the  overall  potential  for  a  shoulder  problem  is  reduced.  This 
is  consistent  with  the  ergonomics  research  literature  which  indicates  that  a  job  designed 
with  task  variety  should  reduce  the  overall  potential  for  WMD  development.  Also,  since 
the  rating  system  still  indicates  that,  when  welding  is  performed,  the  shoulder  is  at  “high” 
risk,  the  technician  can  be  directed  to  identify  controls  which  reduce  exposure  to 
ergonomics  risk  factors  that  impact  the  shoulder  during  welding. 

While  averaging  may  not  always  reflect  the  precise  daily  physical  experience  of  the 
employee,  it  provides  the  technician  with  a  standardized  method  for  determining  the 
impact  of  overall  daily  exposure  and  focusing  problem-solving  efforts  in  order  to  achieve 
the  desired  impact  on  employee  health  and  safety.  This  concept  can  be  referred  to  as 
High-Impact,  Precision-Strike  problem-solving. 

3.5.2  Task  Score.  The  individual  Task  Score  is  determined  by  selecting  the 

highest  numerical  body  region  score  for  that  task.  The  highest  numerical  body  region 
score  is  converted  into  a  high,  medium  or  low  rating.  The  reason:  the  feeling  of  fatigue 
or  pain,  which  are  often  precursors  to  WMD  development,  is  not  “averaged”  throughout 
the  body  by  the  employee.  For  example,  if  exposure  to  a  high  level  of  risk  factors  causes 
an  employee’s  shoulder  to  hurt,  the  employee  does  not  think,  “my  shoulder  hiuls,  but  the 
rest  of  my  body  is  OK,  so  I  must  be  OK.”  Rather,  the  employee  reports  a  shoulder 
problem  because  that  part  of  the  body  hurts.  Therefore,  if  the  shoulder  is  exposed  to  a 
high  level  of  ergonomics  risk  factors,  the  Task  Score  reflects  that  most  significant 
exposure. 


3.5.3  Overall  Job  Priority  Score.  The  Overall  Job  Priority  Score,  high, 

medium,  or  low,  is  determined  by  selecting  the  highest  Body  Region  Priority  Score.  The 
basis  for  this  scoring  concept  is  identical  to  that  which  was  described  in  the  previous 
section.  Overall  Priority  Rating  is  used  to  determine  which  jobs  need  the  most  immediate 
attention. 


3.5.4  Interpretation  and  Intended  Use  of  the  Results.  The  Overall  Job 

Priority  Rating/Score  is  used  to  determine  which  jobs  to  address  first.  Task 
Ratings/Scores  are  used  to  determine  which  task(s)  within  the  job  need  to  be  the  focus  of 
problem-solving  efforts.  And  finally,  the  Body  Region  Scores  for  each  task  are  used  to 
target  the  identification  of  controls  for  the  body  parts  that  are  exposed  to  the  highest  level 
of  ergonomics  hazards.  The  objective  is  precision-strike  focus,  high  impact  results. 
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4.0 


BASIS  FOR  VALIDATING  THE  METHODOLOGY 


4.1  APPROACH 

Testing  of  the  Guide  was  completed  in  two  major  steps:  alpha  testing  and  beta  testing. 

The  purpose  of  alpha  testing  was  to  provide  an  iterative  basis  for  methodology 
development.  Experienced  ergonomists  served  as  test  subjects  to  enhance  the  usability 
and  reliability  of  the  Guide  prior  to  subsequent  testing.  Beta  testing  examined  the 
performance  of  the  Guide  in  a  controlled  enviroiunent  designed  to  be  as  similar  as 
possible  to  actual  use.  Beta  testing  used  non-ergonomists  (Air  Force  personnel)  to 
complete  the  testing.  A  total  of  32  test  job  scenarios  were  developed  containing  video  and 
text  information  for  typical  maintenance  and  inspection  tasks  which  are  of  concern  to  Air 
Force  Ergonomics  Working  Groups.  These  test  job  scenarios  were  used  for  both  alpha 
and  beta  testing. 

Prior  to  conducting  the  alpha  testing,  two  ergonomists  familiar  with  the  jobs  used  in  the 
scenarios  completed  an  independent  assessment.  The  gold  standard  ergonomists  rated  the 
potential  for  WMD’s  to  each  of  the  5  body  areas  for  each  job  scenario.  The  gold  standard 
ergonomists  then  provided  an  overall  rating  for  WMD  potential  for  the  job.  The  ratings 
were  based  on  a  3  point  scale  (low,  medium  or  high)  in  order  to  correspond  directly  to 
methodology  ratings.  The  gold  standard  ratings  were  based  on  professional  judgment  and 
experience  rather  than  outputs  from  any  formal  assessment  protocols.  This  process  was 
chosen  to  avoid  potential  biases  which  could  be  present  in  an  assessment  protocol.  These 
findings  from  the  two  ergonomists  served  as  a  Gold  Standard  to  which  the  methodology 
performance  was  compared.  The  purpose  of  the  Gold  Standard  was  to  provide  an 
indicator  of  concurrent  validity  and  to  suggest  content  changes  to  the  methodology  during 
alpha  testing. 

The  factors  which  were  considered  when  selecting  the  Gold  Standard  ergonomists  were: 

•  experience  in  industrial  ergonomics  training,  workstation  adjustments  or 
modifications,  and  equipment  design; 

•  first-hand  knowledge  of  Air  Force  maintenance  and  inspection  jobs;  and 

•  limited  involvement  in  the  Maintenance  &  Inspection  Level  I  Guide 
development. 

The  ergonomists  selected  as  Gold  Standards  were  J.  Nelson  and  A.  Marcotte. 
Qualification  Summaries  are  provided  in  Appendix  A  Note:  J.  Nelson  also  served  as 
Gold  Standard  on  the  JR/PD  Survey  and  the  Administrative  Guide  development  projects. 
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4.2 


METHOD 


4.2.1  Research  Design.  It  was  estimated  the  five  subjects  would  be  sufficient 
for  the  alpha  test  to  provide  a  preliminary  evaluation  of  the  guide  and  to  provide  guidance 
for  required  changes.  It  was  also  estimated  that  ten  subjects  would  be  sufficient  to 
demonstrate  the  utility  of  the  Guide.  The  beta  testing  was  then  designed  to  use  10  BEF 
technicians  as  subjects.  Each  subject  (five  in  alpha  test,  ten  in  beta  test)  evaluated  each  of 
the  Job  Scenarios.  Alpha  testing  used  32  scenarios,  which  due  to  the  iterative  nature  of 
the  alpha  testing  process  were  not  presented  in  a  fully  random  order.  Beta  testing  used  30 
scenarios  administered  in  random  order  to  control  order  effects.  This  was,  therefore,  a 
repeated  measures  design,  with  multiple  dependent  measures.  The  dependent  measures 
collected  for  each  scenario  were  the  responses  to  each  of  24  Job  Factor  questions,  the 
priority  scores  and  ratings  for  each  of  the  five  body  regions  (shoulder/neck, 
hand/wrist/arm,  back/torso,  legs/feet  and  head/eyes),  the  overall  priority  rating  for  the 
job,  the  corrective  action  selections,  and  time  to  complete  the  evaluation. 

4.2.2  Alpha  Testing.  Alpha  testing  refers  to  a  series  of  three  iterative  trials 
used  in  the  development  process  of  the  Maintenance  &  Inspection  methodology.  Each 
step  was  designed  to  provide  feedback  regarding  different  aspects  of  the  guide 
development.  Alpha  step  one  involved  benchmarking  the  assessment  usability  against  4 
methods  which  varied  considerably  in  their  approaches  (OSHA  draft  checklist  [1],  ANSI 
checklist  [2],  PLIBEL  [3]  and  RULA  [8]).  Alpha  step  two  investigated  the  usability  of 
the  case  studies  for  selecting  solutions.  Alpha  step  three  previewed  the  beta  test, 
investigating  the  reliability  and  validity  of  the  Guide. 

4.2.2. 1  Subjects.  Six  ADL/TJI  ergonomists  participated  in  alpha  steps  one  and 
two.  Five  ADL/TJI  ergonomists  participated  in  the  third  step  of  alpha  testing.  These 
ergonomists  are  experienced  in  the  use  of  assessment  tools  and  ergonomics  checklists. 

4.2.2.2  Procedure  and  Apparatus. 

4.2.2.2.1  Step  One— Assessment  Benchmarking.  Each  of  the  ergonomists  was 

provided  with  the  following  materials: 

•  5  Job  Scenarios  and  corresponding  video  tape; 

•  Level  I  Ergonomics  Assessment  Checklist  (draft  1)  and  Scoring  Summary 
(see  appendix  F); 

•  OSHA  Draft  Checklist; 

•  ANSI  Draft  Checklist; 

•  RULA; 

•  PLIBEL;  and 

•  Usability  questionnaires. 
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The  Job  Scenarios  used  in  this  step  were  specifically  selected  to  represent  a  variety  of 
situations  and  exposures  to  ergonomics  hazards.  Tasks  included  static  efforts,  forceful 
efforts,  seated  work,  standing  work,  upper  extremity  intensive  tasks,  and  whole  body 
activities.  The  jobs  selected  included  tasks  of  a  highly  repetitive  nature  and  ones  with 
more  variety  within  the  task  steps.  The  ergonomists  completed  each  of  the  5  assessment 
methods  for  as  many  of  the  Job  Scenarios  as  possible  within  a  4-hour  time  period.  At  the 
end  of  this  session,  each  ergonomist  completed  an  usability  questiormaire  for  each 
method.  After  the  usability  results  were  tabulated,  one  ergonomist  lead  a  focus  group 
sessions  soliciting  feedback  for  methodology  enhancements. 

Although  the  sample  group  was  clearly  biased  (they  were  not  blind  to  the  methods),  the 
results  suggested  that  the  Level  1  methodology  and  the  RULA  were  considered  more 
useable  than  the  OSHA,  ANSI  and  Plibel  methods.  These  two  methods  shared  the  top 
two  spots  for  nearly  every  question.  The  RULA  and  ANSI  were  considered  the  two  tools 
which  were  most  geared  towards  ergonomists  (rather  than  non-ergonomists).  The  ANSI 
method  was  noted  to  have  a  comprehensive  definition  of  risk  factors,  although  the 
ergonomists  felt  that  it  would  be  difficult  for  a  novice  to  use.  The  OSHA  method  was 
considered  to  be  the  easiest  method  for  a  novice  to  use.  The  ergonomists  felt  that  the 
RULA  was  clear  and  concise,  collecting  considerable  information  with  a  small  amount  of 
paper.  The  identified  strong  points  of  the  Level  1  methodology  were  the  clarity  of 
questions,  clarity  of  risk  factors  (including  ease  of  estimation),  usefulness  for  generating 
solutions  and  interpretation  of  scores.  Areas  identified  for  improvement  included  the 
treatment  of  static  postures  (the  ergonomists  felt  that  these  were  under-reported  by  the 
tool)  and  the  length  of  the  checklist. 

4.2.2.2.2  Step  Two— Solution  Selection  Usability.  Step  two  occurred  in 

conjimction  with  step  one.  Two  days  after  completing  step  one,  the  ADL/TJT  ergonomists 
were  provided  with: 

•  The  same  five  Job  Scenarios  and  corresponding  video  tapes; 

•  Their  completed  Level  I  Ergonomics  Assessment  Checklists  (draft  1)  and 
Scoring  Summaries  (see  Appendix  F); 

•  Case  Study  Manual  (draft  1),  and; 

•  User’s  Instructions  (draft  1). 

The  ergonomists  reviewed  the  video  tapes  and  their  responses  to  the  Ergonomics 
Assessment  Checklists.  Based  on  this  information,  the  ergonomists  selected  solutions 
Jfrom  the  case  study  manual.  Each  ergonomist  selected  solutions  firom  the  case  study 
manual  for  all  jobs  on  which  they  had  completed  an  analysis  earlier  in  the  week.  After 
this  was  completed,  one  ergonomist  led  a  focus  group  session  to  determine  enhancements 
to  the  case  study  manual.  The  Alpha  test  results  are  provided  in  Appendix  B. 


4-3 


4.2.2.23  Step  Three-Reliability  and  Validity.  Each  of  the  alpha  test  subjects  was 
provided  with  the  following  materials. 

•  32  Job  Scenarios  and  corresponding  video  tape; 

•  Level  I  Ergonomics  Assessment  Checklist  (draft  2)(see  Appendix  F); 

•  Checklist  Scoring  Summary  (draft  2); 

•  Corrective  Action  List  (part  of  Checklist  Scoring  Summary); 

•  Case  Study  Problem-Solving  Matrices  (draft  2);  and 

•  User’s  Instructions  (draft  2). 

The  tasks  on  which  the  Job  Scenarios  were  based,  were  selected  to  represent  a  variety  of 
Air  Force  Maintenance  &  Inspection  tasks  and  a  variety  of  risk  factor  exposures.  The  job 
scenarios  are  representative  of  tasks  that  would  typically  be  considered  for  ergonomics 
intervention,  rather  than  all  Maintenance  &  Inspection  tasks.  As  a  result,  each  task  would 
likely  have  some  exposure  to  ergonomics  hazards.  The  Job  Scenarios  present  a 
“standardized”  result  of  the  data  collection  process  to  ensure  that  each  person  using  the 
Guide  bases  their  answers  on  the  same  information.  In  actual  use,  the  data  collection 
process  is  a  critical  component  for  obtaining  results  with  the  Guide.  The  potential  for 
variations  within  this  process  are  not  evaluated.  It  is  expected  that  training  and  experience 
will  yield  consistently  accurate  data  collection. 

The  Level  I  Ergonomics  Assessment  Checklist  was  used  to  analyze  the  job.  The 
Checklist  Scoring  Sununary  was  used  to  score  the  checklist  and  record  results  of  the 
pattern-matching  process  using  the  52  Case  Study  Problem-Solving  Matrices.  User’s 
instructions  were  provided  to  ensure  that  each  ergonomist  followed  the  same  procedure 
for  applying  the  methodology  in  the  Guide.  The  ergonomists  commented  on  the  usability 
of  the  Guide  components  and  user’s  instructions.  A  third  draft  of  each  of  the  Guide 
components  was  developed  to  reflect  those  comments  (refer  to  section  5.4.1,  Usability). 
After  alpha  testing  was  completed,  a  consensus  score  for  each  Job  Factor  question  from 
the  Level  I  Assessment  Checklist  was  developed  to  serve  as  a  testing  standard  during  beta 
testing.  It  was  determined  that  there  was  insufficient  information  to  accurately  answer  or 
evaluate  the  environmental  questions.  Since  these  questions  do  not  figure  in  the  scoring 
and  are  not  reflected  in  the  Case  Studies,  no  analysis  was  conducted  on  these  questions. 

4.23  Beta  Testing. 

4.23.1  Subjects.  Eleven  Air  Force  personnel  were  selected  to  participate  in  a 

single-step  beta  test.  These  personnel  were  to  be  selected  to  “match”  the  targeted  end- 
user  population:  BEF  technician  with  2-3  years  of  experience.  Eleven  personnel  were 
provided  and  the  test  was  conducted  at  Hill  AFB.  The  participants  and  their  experience 
levels  are  described  in  Table  4.2.  Since  beta  testing  was  planned  for  10  persons,  two 
people  were  paired  together  to  complete  testing.  This  was  not  anticipated  to  affect 
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scoring.  One  person  had  previously  completed  the  beta  testing  for  the  administrative 
guide,  which  could  provide  a  small  bias  in  favor  of  better  test  performance. 

The  rationale  for  selecting  ten  as  the  appropriate  number  of  subjects  was  based  on  the 
practicalities  of  conducting  the  research.  The  time  frame  and  the  budget  for  this  project 
permitted  only  ten  subjects'  data  to  be  collected  and  analyzed.  The  time  frame  did  not 
permit  the  alpha  test  data  to  be  completely  analyzed  and  used  as  a  basis  for  determining 
observed  effect  sizes.  The  different  nature  of  the  subjects  in  the  alpha  and  the  beta  tests 
would  have  rendered  any  effect  size  estimates  susjject.  Without  any  reliable  estimate  of 
effect  size,  goal  was  to  run  as  many  subjects  as  possible.  However,  given  the  budget  and 
the  time  frame  available  for  this  project,  the  number  of  test  subjects  appears  to  have  been 
sufficient  to  demonstrate  an  adequate  level  of  reliability,  sensitivity,  and  validity. 

The  broad  range  of  significant  findings  suggests  that  power  was  indeed  adequate  to 
establish  statistically  significant  results.  The  analyst  is  unaware  of  any  power  calculations 
appropriate  for  the  kappa  statistic.  A  post-hoc  assessment  of  power  for  the  60  t-tests 
comparing  beta  test  job  ratings  for  each  scenario  to  both  the  gold  standard  and  consensus 
ratings  was  performed  by  calculating  the  average  effect  size  for  the  comparisons  to  the 
gold  and  the  comparisons  the  consensus.  Based  on  these  two  average  effect  sizes,  the 
power  calculation  for  beta  versus  consensus  t-test  was  .24  and  the  power  for  beta  versus 
gold  standard  was  .74.  The  lower  power  for  the  comparison  to  consensus  ratings  is  due  to 
the  greater  degree  of  agreement  between  consensus  ratings  and  beta  test  ratings.  This 
suggests  that  power  was  adequate  for  the  test  performed. 


4-5 


Table  4.1 

Air  Force  Personnel  for  Beta  Test 


Name 

Organization 

Previous  Ergonomics 
Training 

Experience 

Terefe  Beruk 

74  MEDGP/SGPB 

yes 

2  years 

lara  Hofinann  AlC 

77  MEDGP/SGPB 

no 

2  years 

Jason  Shannon  SrA 

78  AMDS/SGPB 

no 

2  years 

Tracy  Baca  AlC 

75  AMDS/SGPB 

yes 

2  years 

Patricia  Richardson 

75  AMDS/SGPB 

yes 

5  years 

Benjamin  Holley  AlC 

74  AMDS/SGPB 

yes 

2  years 

Thomas  Allen  AlC 

72  AMDS/SGPB 

yes 

2  years 

Altadawn  Bell 

76  MDOS/SGPB 

yes 

2  years 

Joseph  Emery  Ssgt 

75  AMDS/SGPB 

yes 

12  years 

David  DeSmet 

75  AMDS/SGPB 

yes 

12  years 

Mathew  Schwally 

No  further  data  available  | 

Note:  Some  variation  in  results  was  expected  between  personnel  with  previous 
knowledge  of  ergonomics,  particularly  those  who  have  had  previous  training  in 
ergonomics  analysis.  The  test  pool  of  10  subjects  was  anticipated  to  provide  sufficient 
power  for  testing. 

4.23.2  Procedure  and  Apparatus.  Each  of  the  beta  testers  was  provided  with 

the  following  materials: 

•  30  Job  Scenarios  and  corresponding  video  tape  (plus  one  sample); 

•  Level  I  Ergonomics  Assessment  Checklist  (draft  3)  (see  Appendix  F); 

•  Checklist  Scoring  Summary  (draft  3)  (see  Appendix  F); 

•  Corrective  Action  List  (part  of  Checklist  Scoring  Summary); 

•  Case  Study  Problem-Solving  Matrices  (draft  3);  and 

•  User’s  Instructions  (draft  3). 

The  testing  was  conducted  in  the  following  steps:  briefing/overview  of  the  methodology, 
testing,  and  outbriefing/focus  group. 
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4.233  Briefing/Overview  of  the  Guide.  The  ergonomist/facilitator  provided  a 

two-hour  briefing  using  a  sample  job  to  demonstrate  the  methodology,  use  of  the 
materials,  and  process  for  completing  the  assessment  and  pattern-matching  activity.  The 
results  from  the  sample  job  are  not  included  in  the  beta  test  results. 

4.23.4  Testing.  The  testing  process  and  materials  provided  were  the  same  as  for 

the  alpha  test  step  three  (with  the  appropriate  revisions).  Each  beta  tester  followed  the 
User’s  Instructions  to  apply  the  Methodology  to  each  of  the  Job  Scenarios.  The  testing 
process  was  completed  in  7  days.  Each  beta  tester  was  also  asked  to  record  the  amount  of 
time  required  to  complete  both  the  Level  I  Ergonomics  Assessment  Checklist/Checklist 
Scoring  Summary  and  the  pattem-matching/control-identification  process  for  each  Job 
Scenario. 


4.2.3.5  Outbriefing/Focus  Group.  Information  on  usability  was  obtained  by  the 

Ergonomist  who  administered  the  beta  test.  The  ergonomist  obtained  comments 
throughout  the  testing,  distributed  on  usability  questionnaire  and  conducted  a  focus 
group/out-briefing  with  participants  to  obtain  additional  comments  after  the  testing  was 
complete.  For  more  details  on  the  process  used  to  obtain  usability/practicality  comments, 
refer  to  Section  4.3.3,  Practicality  Testing. 

4.3  DATA  ANALYSIS  AND  SCIENTIFIC  BASIS 

The  purpose  of  the  data  analysis  is  to  show  that  the  design  of  the  Level  I  Guide  meets  the 
project  goals.  To  do  this  it  is  necessary  to  demonstrate  that  the  Guide  is  reliable, 
sensitive,  valid  and  practical.  These  concepts  are  easily  defined.  A  scale  is  said  to  be 
valid  if  it  measures  what  it  is  intended  to  measure.  For  example,  if  one  needs  to  know 
how  much  force  will  be  required  to  provide  a  given  acceleration  to  an  object,  then 
weighing  the  object  provides  a  valid  measure,  unless  it  is  necessary  to  consider  other 
gravitational  systems.  In  that  case,  weight  would  not  be  a  valid  measure  of  an  object’s 
mass.  An  ergonomics  assessment  tool,  such  as  the  Level  I  Checklist,  should  measure 
some  aspect  of  human  health,  comfort  or  performance  in  order  to  be  a  valid  measure. 

Reliability,  on  the  other  hand,  refers  to  the  accuracy  and  the  repeatability  of  the 
measurement  of  a  variable.  The  reliability  of  an  instrument  is  Ae  foundation  for  the  other 
concepts.  An  instrument  must  be  reliable  in  order  to  be  valid,  since  one  can  not  be  sure 
that  the  “valid  results”  obtained  one  time  will  be  repeated.  In  our  weight/mass  example, 
asking  people  to  estimate  the  mass  of  an  object  by  picking  it  up  would  be  a  valid 
measure,  but  would  have  much  less  reliability  than  using  an  electronic  scale.  An 
electronic  scale  would  be  very  reliable,  having  excellent  repeatability  and  accxiracy. 
However,  if  the  goal  was  to  evaluate  the  beauty  of  an  object,  rather  than  its  mass,  then  the 
scale  would  not  be  valid,  even  though  it  would  be  highly  reliable. 
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Sensitivity  refers  to  how  well  the  scaling  levels  of  a  tool  reflect  the  levels  that  exist. 
Continuing  our  weight/mass  example,  if  the  electronic  scale  measured  (with  reliability 
and  validity)  the  weight  of  objects  in  poimds,  but  you  were  interested  in  differences  in 
objects  weighing  less  than  ounce,  then  the  scale  would  not  have  sufficient  sensitivity  to 
identify  the  differences.  In  order  to  have  sufficient  sensitivity,  one  would  need  a  scale  that 
reported  much  finer  increments. 

Practicality  identifies  the  usefulness  and  usability  of  an  instrument.  Continuing  with  our 
weight/mass  example,  if  our  electronic  scale  reported  weights  with  sufficient  sensitivity, 
it  might  still  need  to  meet  constraints  such  as  speed  of  calibration  and  portability  in  order 
to  be  a  practical  scale  for  its  intended  application. 

Data  analysis  was  accomplished  using  Microsoft®  Excel  v7.0  and  SAS  v6.1 1. 

4.3.1  Reliability  Testing.  Reliability  testing  of  assessment  tools  generally  takes 

one  of  two  forms,  test/re-test  reliability  and  inter-rater  reliability.  Test/re-test  reliability 
defines  how  well  the  same  person  will  achieve  the  same  results  using  a  tool  at  different 
times.  Inter-rater  reliability  defines  how  well  different  people  will  agree  on  the  results. 

Several  techniques  have  been  used  and  reported  for  inter-rater  reliability  testing  with 
ergonomic  assessment  tools.  The  coefficient  of  variation  was  used  in  one  study  [10]  with 
scores  of  less  than  20  percent  for  most  measures.  Multiple  regression  has  also  been  used 
[10]  with  a  finding  of  no  statistically  significant  differences  between  raters  supporting 
reliability.  Kemmlert  [3]  found  weighted  averages  of  Kappa  ranging  from  0.24  to  0.44 
and  a  percent  agreement  often  above  70  percent.  Keyserling,  Stetson,  Silverstein  and 
Brouwer  [9]  used  inter-rater  agreement  percentages  as  a  part  of  their  validation.  There  is 
little  consensus  on  the  best  methods  for  demonstrating  inter-observer  agreement  (Meister, 
1985  [47]). 

Individual  questions  were  evaluated  for  reliability  by  calculating  Kappa  statistics  for  the 
alpha  and  beta  tests  (Section  4.2.2  and  4.2.3).  The  Kappa  statistics  were  chosen  because 
it  represents  the  most  accurate  method  for  testing  reliability  while  controlling  for  the 
effects  of  chance.  The  data  meet  all  of  the  assumptions  for  Kappa  (Cohen,  1960  [48]; 
Breiman  &  Prediger,  1981  [49])  which  are  that : 

•  the  objects  categorized  are  independent; 

•  the  raters  operate  independently;  and 

•  the  categories  are  independent,  mutually  exclusive,  and  exhaustive. 

Since  it  would  be  expected  that  a  certain  amount  of  agreement  would  occur  by  chance, 
like  having  50%  correct  on  a  true/false  test,  the  Kappa  statistic  reports  agreement  after 
chance  has  been  removed.  A  Kappa  value  can  be  interpreted  as  a  percent  of  agreement. 
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for  instance  a  Kappa  of  0.75  indicates  an  agreement  rate  of  75%  after  chance  has  been 
removed. 

For  interpreting  the  Kappa  values  the  following  interpretations  were  used,  consistent  with 
those  suggested  by  Landis  and  Koch  (1977)  [50]  and  as  shown  in  Table  4.2: 

Table  4.2 

Kappa  Value  Interpretation 


Kappa 

Values 

Interpretation 

0.81  to  1.0 
0.61  to  0.80 
0.41  to  0.60 
0.21  to  0.40 
0.0  to  0.20 
<0.0 

Almost  Perfect 
Substantial 

Moderate 

Fair 

Slight 

Poor 

The  scores  (item  level,  body  region  level,  and  risk  rating)  from  both  the  alpha  and  beta 
test  sessions  were  analyzed  independently  using  the  Kappa  statistic. 

4.3.2  Sensitivity  Testing.  Measures  of  tool  sensitivity  have  not  generally  been 

reported  in  the  ergonomics  assessment  tool  literature.  Sensitivity  is  demonstrated  by  how 
well  the  tools  can  distinguish  risk  factor  levels  from  one  job  to  another.  Discriminant 
analysis  was  selected  as  the  primary  method  for  the  sensitivity  analysis.  Discriminant 
analysis  is  a  multivariate  technique  that  uses  both  grouping  data  (in  this  case,  the  either 
the  Gold  Standard  or  the  Consensus  judgment  of  the  risk  level  of  each  scenario)  and  the 
measured  variables,  and  constructs  linear  combinations  of  the  measured  variables  to 
create  the  best  possible  categorization  to  match  the  known  grouping.  The  discriminant 
analysis  can  be  conducted  many  ways;  simultaneous  and  stepwise  analyses  were 
conducted  for  this  project.  One  major  advantage  for  iise  of  discriminant  analysis  is  that, 
if  only  the  statistically  significant  discriminant  functions  are  used,  then  it  is  more  likely  to 
find  increased  sampling  stability  than  if  all  of  the  data  was  used  for  the  “best” 
classification  (Tatsuoka,  1974  [51]).  For  this  analysis,  two  variables  were  constructed  for 
each  scenario.  The  scenario  was  rated  as  High,  Medium  or  Low  Risk  based  on  the  Gold 
Standard  evaluation  (the  Gold  variable)  and  similarly  rated  as  High,  Medium  or  Low 
Risk  based  on  the  Consensus  judgment  (the  Consensus  variable).  Each  of  these 
categorizations  were  used  as  the  grouping  or  classification  variable  for  discriminant 
analysis.  The  subject's  Overall  Job  Priority  Rating  and  the  five  Body  Region  Priority 
Scores  (Shoulder/Neck,  Hands/Wrists/Arms,  Back/Torso,  Legs/Feet,  and  Head/Eyes) 
were  used  as  the  predictor  variables. 
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4.3.2.1  Validity.  A  variety  of  methods  have  been  used  to  establish  the  validity  of 
ergonomic  assessment  procedures.  Predictive  validity  has  been  suggested  through 
comparisons  with  incidence  rate  [7]  and  with  discomfort  ratings  [8].  Any  predictive 
validity  measures  for  simple  assessment  techniques  are  subject  to  confoimding  variables 
which  could  greatly  reduce  power.  Concurrent  validity  has  been  supported  through 
comparing  expert  ratings  on  two  different  techniques  [3]  and  by  comparing  novice  users 
of  a  simple  technique  with  expert  users  of  a  more  detailed  methodology  [9].  Content 
validity  has  been  reported  as  an  item  match  with  scientific  literature  [3]. 

4.3.2.1.1  Content  Validity.  The  content  validity  was  based  on  using  referenced 
criteria  to  select  questions  during  tool  development.  Kemmlert  [3]  reports  content 
validity  for  the  PLIBEL  method.  Scientific  literature  was  reviewed  to  provide  references 
for  each  item  in  PLIBEL.  The  primary  approach  for  assessing  content  validity  for  the 
Level  I  Checklist  was  to  provide  support  in  the  literature  (particularly  validated 
assessment  tools)  for  all  Job  Factor  questions  and  scaling  approaches. 

4.3.2.1.2  Concurrent  Validity.  The  primary  method  for  assessing  the  concurrent 
validity  of  the  evaluation  was  to  compare  the  test  evaluations  with  the  Gold  Standard. 

The  Gold  Standard  results  were  compared  to  the  alpha  and  beta  results  iising  t-tests.  The 
t-test  was  selected  because  it  is  known  to  be  robust  to  violations  of  most  of  the 
distributional  assumptions.  With  only  5  alpha  and  10  beta  subjects,  it  is  unlikely  that  the 
data  will  meet  the  assumptions  of  the  test.  However,  the  test  still  provides  a  reliable 
indication  of  the  differences  between  the  alpha  or  beta  test  scores  and  the  Gold  Standard. 
Furthermore,  because  most  assumption  violations  would  cause  the  test  to  be  liberal,  in 
this  particular  case  that  would  work  against  the  demonstration  of  agreement  between  the 
test  scores  and  the  Gold  Standard.  The  data  analyst  is  unaware  of  any  statistic  that  would 
be  more  accurate  in  describing  this  particular  situation  of  comparing  a  set  of  “correct” 
answers  with  data  collected  from  a  number  of  subjects  on  each  of  the  30  Scenarios.  The 
t-test  is  simple,  well  known,  well  imderstood,  and  easily  communicated. 

4.3.2.1.3  Predictive  Validity.  Predictive  validity  was  not  assessed  because  of 
much  of  the  discomfort  data  was  lacking  or  suspect. 

4.3.3  Practicality  Testing.  While  the  practicality  of  the  methodology  is 
perhaps  the  most  important  consideration,  it  is  also  the  most  difficult  to  quantify  in  a 
short-term  test.  The  solid  test  of  practicality  is  how  frequently  the  tool  is  applied  in 
examining  workstations  and  the  results  obtained  through  the  changes.  In  the  shorter- 
term,  several  measures  were  used  to  examine  the  practicality  of  the  tool.  The  usability 
comments  were  tabulated,  Avith  the  assumption  that  a  highly  useable  tool  is  more  practical 
than  other  tools.  The  descriptive  statistics  of  time  requirements  were  compared  to  the 
criteria  established,  with  the  assumption  that  an  effective  tool  that  can  be  completed 
within  the  time  suggestions  has  some  practicality.  The  overall  agreement  regarding 
solution  options  was  calculated  and  reported,  based  on  the  assumption  that  a  method 
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which  leads  to  consistent  solution  recommendations  is  practical.  While  none  of  these 
measures  prove  practicality,  positive  results  in  all  these  areas  would  suggest  that 
technicians  will  find  that  the  tool  is  a  practical  one  to  use. 

4.3.4  Data  Reduction.  Data  were  entered  into  an  Excel  spreadsheet  where 
checking  and  coding  were  conducted.  The  data  were  written  to  an  ASCII  file  and 
modified  using  a  text  word  processor  for  input  to  SAS.  Three  SAS  datasets  were  created, 
one  for  each  of  the  alpha  and  beta  samples  with  one  logical  record  per  subject,  and  one 
for  the  beta  sample  with  30  logical  records  per  subject  (one  record  for  each  Scenario). 

The  single  record  per  subject  datasets  were  used  for  the  reliability  analyses,  and  the 
multiple  record  per  subject  dataset  were  used  for  the  discriminate  analysis.  No  multiple 
record  per  subject  dataset  was  generated  for  the  alpha  test  because  it  was  judged  that  there 
were  too  few  subjects  (5)  to  permit  a  meaningful  discriminate  analysis. 

4.3.5  Summary  of  Statistical  Selection  Rationale.  Preliminary  data  screening 
was  not  conducted  to  evaluate  variable  distributions.  For  most  parametric  statistics,  the 
distributional  tests  for  normality,  etc.,  are  more  sensitive  than  are  the  parametric  tests 
themselves.  The  parametric  tests  are  quite  robust  to  violations  of  the  distributional 
assumptions,  especially  if  there  are  equal  number  of  subjects  in  each  group.  Furthermore, 
the  design  ensured  that  the  assumptions  of  the  primary  statistic  for  reliability  and  validity 
were  met.  The  data  meet  all  of  the  assumptions  for  Kappa  [48,49],  which  are  that: 

•  the  objects  categorized  are  independent; 

•  the  raters  operate  independently;  and 

•  the  categories  are  independent,  mutually  exclusive,  and  exhaustive. 

As  discussed  above,  much  of  the  data  collected  for  this  project  was  recoded  to  be  on  a 
High,  Medium  or  Low  scale.  These  data  therefore  are  scaled  as  at  least  ordinal  data, 
because  we  know  that  High  is  larger  than  Medium,  and  Medium  is  larger  than  Low. 
However,  while  the  scale  itself  is  only  a  three  level  scale,  clearly  the  data  represented 
have  more  range  than  that.  Furthermore,  the  original  data  can  lie  on  a  continuum 
between  High  and  Low.  Therefore,  for  the  purposes  of  illustrating  the  observed  data, 
some  of  the  exploratory  type  of  analyses,  means  and  standard  errors  were  calculated, 
graphed  and  used  in  analyses.  In  many  cases,  this  resulted  in  values  intermediate 
between  High,  Medium  and  Low  levels  being  reported  the  reader  is  urged  to  regard  these 
as  approximating  the  central  tendency  and  variability  of  the  true  rmderlying  variable. 
Importantly,  the  critical  analyses,  the  Kappa  analyses  and  the  discriminant  analyses,  used 
these  data  as  representing  a  categorical  scale. 

This  research  project  was  designed  to  use  the  power  of  a  repeated  measures  design  to 
provide  as  much  value-added  information  as  possible  on  the  Level  I  Guide.  As  with  any 
other  research  project,  additional  testing  with  more  subjects,  on  additional  Scenarios 
would  yield  increased  precision,  with  a  commensurate  increase  in  cost  and  testing  time. 
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The  selection  of  a  data  analysis  strategy  is  a  process  that  depends  on  may  factors.  Minor 
changes  in  the  research  methodology,  designed  to  improve  the  precision  of  the  study,  can 
necessitate  substantial  changes  in  the  analysis.  The  data  analyses  for  this  project  were 
finalized  after  the  research  methodology  was  fully  developed,  following  extensive 
consultation  with  senior  academic  research  scientists  to  provide  and  accurate,  concise, 
and  clear  portrayal  of  the  results.  It  is  the  judgment  of  the  analyst  that  the  value-added 
approach  and  results  provide  an  adequate  initial  basis  for  evaluating  the  utility  and 
practicality  of  the  Level  I  Guide.  The  raw  data  is  provided  in  Appendix  E. 
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5.0 


RESULTS 


5.1  RELIABILITY 

The  reliability  results  from  the  alpha  and  beta  tests  are  consistent  The  average 
agreements  were  over  60%  (alpha=64%,  beta=67%).  The  mean  value  of  Kappa  for 
individual  questions  is  essentially  identical  between  the  test  groups  (alpha  =  0.29,  beta  = 
0.26).  The  rates  compare  well  with  the  range  of  weighted  averages  of  Kappa  (0.24  to 
0.44)  found  by  KenWert  [3]  and  those  found  in  the  administrative  Guide  development 
(0.22  to  0.23).  These  results  indicate  a  consistent  and  statistically  significant  agreement 
on  individual  questions  (this  agreement  is  classified  in  the  slight  to  fair  range).  The 
ergonomists  and  the  technicians  obtained  similar  levels  of  agreement  on  individual 
questions  as  have  been  found  in  research  studies  using  other  ergonomics  assessment 
tools.  Furthermore,  the  ergonomists  and  technicians  demonstrated  agreement  on  the 
overall  scores  for  the  jobs. 

5.1.1  Item  Level  Agreement.  The  percent  agreement  and  kappa  value  were 
calculated  for  each  job  factor  question  on  the  alpha  and  beta  tests.  Bartko  &  Carpenter 
[48]  recommend  the  “generalized”  kappa  in  cases  where  there  are  polychotomous  data 
and  more  than  two  raters.  Since  both  the  alpha  and  beta  test  item  level  data  fit  this 
situation,  the  generalized  kappa  is  reported  for  item  agreement.  The  percent  agreement 
reported  for  the  alpha  and  beta  tests  is  based  on  the  formula  for  determining  “percent 
observed”  within  the  generalized  kappa  model  [48].  The  95%  Upper  and  Lower 
Confidence  Intervals  are  reported  for  each  question.  The  response  rates  for  scored 
responses  (scores  greater  than  zero)  are  also  reported. 

5.1.1.1  Alpha  Test.  The  general  level  of  agreement  between  ergonomists  using 
the  Level  I  Checklist  is  presented  in  Table  5.1.  The  agreement  rates,  not  corrected  for 
chance,  for  the  alpha  test  questions  ranged  between  35%  and  89%  for  those  questions 
with  sufficient  response  rates  of  risk  factor  presence.  The  utility  of  questions  with  lower 
than  a  10%  response  rate  (30  responses)  would  require  careful  consideration,  particularly 
if  the  question  also  displayed  low  agreement.  While  questions  4, 17, 19, 21, 25, 28  and 
29  had  low  response  rates,  none  of  these  questions  also  displayed  low  agreement.  No 
questions  fit  this  pattern.  Kemmlert  [3]  found  agreement  Kappa  values  in  the  fair  to 
moderate  range  for  individual  questions.  A  total  of  26  of  31  questions  had  similar 
agreement  rates  in  this  study. 

5.1.1.2  Beta  Test.  Questions  with  response  rates  ofless  than  5%  (30  responses) 
were  interpreted  cautiously,  and  considered  for  removal  from  the  Level  I  Checklist  due  to 
the  apparent  limited  applicability  of  the  question  to  typical  maintenance/inspection  jobs. 
However,  the  ergonomists  decided  that  the  question  should  remain  because  these  Job 
Factors  although  infrequent,  are  important  considerations.  To  remove  the  items  at  this 
time  wovild  potentially  limit  the  applicability  of  the  tool  to  a  wide  range  of  work 
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situations  that  may  not  have  been  observed  during  the  site  visits.  Questions  with  low 
response  rates  were  numbers  17, 18  and  22.  Table  5.2  shows  results  for  the  beta  tests. 

A  total  of  16  of  26  questions  fell  in,  or  above,  the  range  of  Kappa  values  foimd  in  the 
checklist  study  by  Kemmlert  [3].  The  questions  which  fell  in  this  range  are  numbers  la, 
lb,  2, 3, 6,  7, 8,  9, 15, 16a,  16b,  18, 19, 21, 22,  and  23.  Nearly  all  questions  fall  in  the 
descriptions  of  slight  to  fair  agreement  [39].  Although  no  questions  met  the  goal  criteria 
for  Kappa  of  0.6  to  0.7,  nearly  all  the  questions  still  have  statistically  significant 
agreement. 

It  appears  that  some  of  the  disagreement  may  be  the  result  of  difficulties  distinguishing 
between  job  fectors  occurring  sometimes  or  frequently.  The  scoring  of  question  la  was 
altered  between  alpha  and  beta  test  administrations  such  that  (for  moderate  duration 
tasks)  sometimes  and  frequently  responses  received  the  same  score.  The  agreement  rose 
between  the  alpha  and  beta  test  from  41%  (Kappa=.17)  to  61%  (Kappa— .34).  Since  the 
results  of  this  Guide  are  being  compared  against  Kemmlert  [3],  which  uses  only  a 
“yes/no”  decision  for  risk  factor  presence,  the  potential  effects  of  this  difficulty  in 
distinguishing  the  frequency  of  occurrence  are  that  this  Guide  will  not  perform  as  well  as 
it  might  otherwise.  The  current  data  coding,  which  is  by  scores,  not  categories,  does  not 
allow  this  comparison  without  re-entering  all  raw  data  from  the  original  data  sheets. 

What  is  important  to  note  is  the  results  from  the  alpha  and  beta  tests  are  consistent.  The 
mean  value  of  Kappa  for  each  is  essentially  identical  (alpha  =  0.29,  beta  =  0.26).  The 
rates  compare  well  with  the  range  of  weighted  averages  of  Kappa  (0.24  to  0.44)  found  by 
Kemmlert  [3].  These  results  indicate  a  consistent,  and  statistically  significant,  slight  to 
fair  agreement. 

5.1.2  Overall  Score  Agreement. 

5.1.2.1  Alpha  Test.  The  overall  reliability  of  the  Level  I  Checklist  scores  and 
ratings  from  the  alpha  test  session  was  measured  using  a  generalized  Kappa  statistic  [52] 
calculated  for  the  Overall  Priority  Rating.  The  agreement  rate  was  62%,  Kappa=  0.18, 
(95%LCI  =  0.00, 95%UCI=  0.36).  This  indicates  a  slight  agreement  (after  chance  is 
removed)  which  is  not  statistically  significant. 

5.1.2.2  Beta  Test.  The  same  statistics  used  in  the  alpha  test  phase  were  calculated 
for  the  beta  test  data.  The  results  were  similar,  although  the  larger  sample  size  helped  to 
produce  statistically  significant  results.  The  agreement  rate  was  55%,  Kappa=.18 
(95%LCI=.10, 95%UCI=.26). 


Statistical  significance  for  agreement  in  the  beta  test  phases  indicates  that  the  agreement 
obtained  was  better  than  would  have  been  expected  due  to  chance.  Practical  significance 
suggests  that  the  reliability  agreement  is  slight,  with  chance  corrected  agreement  rates 
between  10%  and  26%. 
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Table  5.1 

Testing  of  the  Item  Agreement  Among  Ergonomists  for  Alpha  Test 


Question 

Percent 

Kappa 

95% 

LCI 

95% 

UCI 

Non-Zero 

Observations 

la 

41% 

0.17 

0.11 

0.23 

269 

lb 

44% 

0.21 

0.14 

0.28 

159 

2 

55% 

0.29 

0.21 

0.37 

142 

3 

72% 

0.34 

0.18 

0.49 

67 

4 

85% 

0.36 

0.11 

0.61 

27 

5 

37% 

0.24 

0.20 

0.28 

266 

6 

45% 

0.25 

0.19 

0.30 

271 

7 

61% 

0.29 

0.19 

0.40 

109 

8 

65% 

0.24 

0.10 

0.38 

80 

9 

44% 

0.15 

0.08 

0.22 

157 

10 

47% 

0.12 

0.02 

0.21 

123 

11 

61% 

0.24 

0.11 

0.37 

90 

12 

51% 

0.22 

0.13 

0.30 

137 

13 

73% 

0.45 

0.32 

0.58 

88 

14 

83% 

0.36 

0.13 

0.59 

33 

15 

35% 

0.21 

0.17 

0.25 

241 

16 

42% 

0.15 

0.09 

0.21 

194 

17 

84% 

0.36 

0.10 

0.61 

26 

18 

62% 

0.21 

0.08 

0.35 

82 

19 

89% 

0.36 

0.04 

0.69 

11 

20a 

68% 

0.31 

0.17 

0.45 

78 

20b 

81% 

0.35 

0.12 

0.57 

35 

21 

85% 

0.38 

0.12 

0.63 

26 

22 

64% 

0.44 

0.37 

0.52 

271 

23 

54% 

0.19 

0.09 

0.29 

114 

24 

57% 

0.28 

0.18 

0.38 

119 

25 

85% 

0.34 

0.08 

0.61 

23 

26 

66% 

0.36 

0.24 

0.47 

99 

27 

57% 

0.28 

0.19 

0.37 

125 

28 

93% 

0.46 

0.08 

0.85 

2 

29 

93% 

0.35 

-0.09 

0.79 

4 
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Table  5.2 

Testing  of  the  Item  Agreement  Among  End  Users  (Beta  Testers) 


Question 

Percent 

Kappa 

95% 

LCI 

95% 

UCI 

Non-Zero 

Observations 

la 

61% 

0.34 

0.27 

0.40 

483 

lb 

58% 

0.26 

0.18 

0.34 

217 

2 

59% 

0.32 

0.25 

0.38 

244 

3 

74% 

0.34 

0.22 

0.46 

138 

4 

34% 

0.11 

0.08 

0.15 

513 

5 

39% 

0.15 

0.12 

0.18 

448 

6 

79% 

0.43 

0.31 

0.55 

130 

7 

70% 

0.39 

0.31 

0.47 

200 

8 

47% 

0.28 

0.25 

0.31 

382 

9 

63% 

0.42 

0.37 

0.47 

276 

10 

58% 

0.13 

0.03 

0.22 

176 

11 

88% 

0.05 

-0.24 

0.34 

37 

12 

51% 

0.14 

0.07 

0.20 

233 

13 

63% 

0.08 

-0.04 

0.20 

142 

14 

84% 

0.17 

-0.06 

0.39 

59 

15 

50% 

0.25 

0.20 

0.29 

486 

16a 

83% 

0.28 

0.09 

0.47 

73 

16b 

92% 

0.26 

-0.05 

0.58 

31 

17 

95% 

0.14 

-0.33 

0.62 

14 

18 

96% 

0.48 

0.09 

0.87 

21 

19 

62% 

0.41 

0.36 

0.46 

471 

20 

55% 

0.19 

0.11 

0.26 

215 

21 

63% 

0.36 

0.29 

0.42 

235 

22 

96% 

0.40 

-0.02 

0.82 

18 

23 

73% 

0.32 

0.20 

0.44 

136 

24 

53% 

0.07 

-0.01 

0.16 

192 

5.1.2.3  Alpha  -  Beta  Comparison.  The  alpha  and  beta  test  are  not  directly 

comparable  because  of  scoring  changes  that  were  implemented  at  the  conclusion  of  the 
alpha  testing.  Alpha  test  results  suggested  that  the  methodology  may  have  been 
overrating  the  risk  level,  particularly  in  the  hands/wnsts  and  shoulder/neck  body  areas. 
Subjective  feedback  also  indicated  some  confusion  regarding  the  proper  response  to  the 
questions  regarding  static  and  repeated  gripping.  The  individual  job  factor  response 
weightings  were  altered  to  produce  lower  scores,  particularly  for  hands/wnst  and 
shoulder/neck  body  areas.  Several  questions  were  combined  to  both  reduce  the  risk 
ratings  for  these  body  areas  and  to  reduce  potential  response  confusion.  For  example, 
the  separate  questions  on  static  and  repeated  gripping  which  were  present  in  the  alpha 
testing,  were  combined  into  one  question  on  the  beta  testing.  Because  of  these  changes. 
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even  if  the  beta  test  subjects  always  identified  the  same  job  factors  as  the  alpha  test 
subjects  the  body  area  risk  ratings  and  overall  priority  score  would  often  be  different. 

5.2  SENSITIVITY 

5.2.1  Discriminant  Analysis.  Four  discriminant  analyses  were  conducted  for 
this  analysis.  The  analyses  conducted  were: 

•  Stepwise  discriminant  Analysis  -  Gold  rating  of  job 

•  Simultaneous  discriminant  Analysis  -  Gold  rating  of  job 

•  Stepwise  discriminant  Analysis  -  Consensus  rating  of  job 

•  Simultaneous  discriminant  Analysis  -  Consensus  rating  of  job 

5.2.2  Gold  Variable.  The  stepwise  discriminant  analysis  resulted  in  one 
significant  discriminant  function  (F^  sgg  =  9.56,  p  <0.001),  and  used  the  shoulder/neck 
rating  (SNRATE),  the  hand/wrist/arm  rating  (HWARATE)  and  the  back/torso  rating 
(BTRATE)  scores  as  predictors.  This  significant  function  accormted  for  almost  18%  of 
the  variance  in  the  GOLD  rating  of  the  job.  The  simultaneous  discriminant  analysis  using 
all  6  variables  also  resulted  in  one  significant  discriminant  function  (F,2,5g2  =  5.17,p 
<0.001).  The  significant  variables  in  the  simultaneous  discriminant  analysis  were  the 
SNRATE,  the  HWARATE  and  the  BTRATE  scores,  as  in  the  stepwise  analysis,  along 
with  the  overall  job  rating  (JOBRATE)  variable.  The  JOBRATE  variable  was  not 
included  in  the  stepwise  analysis,  because  it  was  highly  correlated  with  SNRATE,  and  the 
correlated  variance  was  important  in  predicting  the  Gold  Standard  classification  of  the 
scenarios. 


5.2.3  Consensus  Variable.  The  stepwise  discriminant  analysis  resulted  in  one 
significant  discriminant  function  (F4  294  =  42.55,  p  <0.001),  and  used  the  SNRATE,  the 
JOBRATE,  the  legs/feet  rating  (LF^TE)  and  the  BTRATE  scores  as  predictors.  This 
significant  fimction  accoimted  for  almost  40%  of  the  variance  of  the  CONSENSUS  rating 
of  the  job.  The  simultaneous  discriminant  analysis  using  all  6  variables  also  resulted  in 
one  significant  discriminant  function  (P(,:i92  ~  28.52, p  <0.001).  The  significant  variables 
in  the  simultaneous  discriminant  analysis  were  the  SNRATE,  the  JOBRATE,  the 
LFRATE  and  the  BTRATE  scores,  as  in  the  stepwise  analysis,  and  the  HWARATE 
variable.  The  head/eyes  rating  (HERATE)  variable  was  not  significantly  related  to  the 
CONSENSUS  rating  of  the  job. 

5.2.4  Discriminant  Analysis  Discussion.  These  four  analyses  point  to  two 
clear  conclusions.  First,  the  Checklist  Tool  clearly  predicts  the  CONSENSUS  results, 
accounting  for  40%  of  the  variance,  while  it  predicts  the  GOLD  variable  results  less  well, 
only  accounting  for  18%  of  the  variance.  Second,  the  major  part  of  this  relationship  is 
due  to  the  correlation  between  the  CONSENSUS  variable  and  the  JOBRATE  and 
SNRATE  variables  (which  are  themselves  highly  correlated,  see  Table  5.3). 
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Table  53 

Beta  Test  Correlation  Matrix  for  Job  Factor  Ratings 


Non-significant  correlations  are  enclosed  in  heavy  boxes 


Variable 
correlation  p 

SNRATE 

HWARATE 

BTRATE 

LFRATE 

HERATE 

JOBRATE 

GOLD 

SNRATE 

1.00 

0.0 

HWARATE 

.628 

1.00 

.001 

0.0 

BTRATE 

.449 

1.00 

.001 

0.0 

LFRATE 

.379 

.414 

1.00 

.001 

.001 

0.0 

HERATE 

-.031 

.147 

1.00 

1.121  1 

.592 

.011 

0.0 

JOBRATE 

.757 

.548 

.117 

1.00 

bS9HI 

0.0 

GOLD 

.069 

-.007 

.275 

1.00 

BlHi 

.231 

.909 

.001 

0.0 

CONSENSUS 

.426 

.350 

.545 

191 

.001 

_ 1 

The  conclusion  that  can  be  drawn  from  these  analyses  is  that  there  is  a  significant 
relationship  between  the  evaluations  fix)m  the  Air  Force  technicians  and  the  expert 
ergonomists  (using  both  the  same  Level  I  Guide  or  an  independent  method),  and  that  the 
Level  I  Guide  can  significantly  and  correctly  distinguish  between  scenarios  that  were 
judged  by  the  experts  to  be  of  high  or  medium  risk  for  ergonomic  risk  factor  exposure. 

5.3  VALIDITY 

53.1  Content  Validity.  There  is  a  strong  scientific  basis  for  the  inclusion  of  all 

the  job  factor  questions  present  in  this  analysis.  All  job  factor  questions  were  derived 
from  peer  reviewed  journals,  established  technical  books,  or  proposed  standards.  Each 
source  has  been  identified  as  a  potential  WMD  risk  factor  and/or  cause  of  localized 
fatigue  and  discomfort  (see  Section  6  for  a  complete  listing  of  references). 

The  list  of  job  factor  questions  has  been  judged  to  be  complete  for  maintenance  and 
inspection  tasks.  At  least  32  individual  analysis  methods  were  evaluated  as  a  part  of  the 
literature  review  process.  All  risk  factor-based  questions  identified  in  the  32  methods 
were  considered  for  inclusion  in  order  to  ensure  that  the  list  of  job  factor  questions  was 
complete. 
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The  question/response  structure  (i.e.,  postural  deviations  or  forces  over  a  period  of 
exposure)  is  based  on  a  well-established  model  of  how  damage  accumulates  (see  Section 
1.3. 2.3).  Several  existing  analysis  methods  [36, 10]  which  use  this  basic  model  have 
been  validated. 


5.3.2  Concurrent  Validity.  Table  5.4  illustrates  the  agreement  between  the 

beta  test  technicians’  results  and  the  Gold  Standard.  Table  5.5  illustrates  the  agreement 
between  the  beta  test  technicians  and  the  consensus  ergonomist.  When  the  beta  test  scores 
are  significantly  different  from  both  the  consensus  and  the  gold  (in  the  same  direction)  it 
can  be  assumed  that  this  difference  is  the  result  of  differences  in  expert  assessments  (gold 
and  consensus)  and  novice  assessments.  When  the  beta  test  is  significantly  different  than 
the  gold  but  not  the  consensus  it  can  be  assumed  that  the  results  illustrate  differences  in 
what  the  Guide  and  the  Gold  Standard  find  within  a  job.  When  the  beta  test  scores 
significantly  split  the  difference  between  the  gold  and  consensus  it  can  be  assumed  that 
these  were  Job  Scenarios  on  the  border  of  a  rating  level.  Table  5.6  summarizes  which  Job 
Scenarios  fit  each  of  these  3  possible  differences.  It  is  apparent  that  most  of  the 
disagreements  were  the  result  of  differences  between  the  Guide  and  Gold  Standard 
approach.  This  suggests  that  expert  users  (alpha  test  consensus)  and  novice  users  (beta 
test  subjects)  obtained  similar  results  from  the  Guide,  and  that  these  results  differed  from 
the  Gold  Standard.  Table  5.7  demonstrates  that  the  Guide  risk  ratings  do  not  differ  from 
the  Gold  Standard  in  a  consistent  direction. 

While  the  differences  are  not  in  a  consistent  direction,  the  differences  are  consistent. 
Investigating  the  specific  aspects  of  the  Job  Scenarios  which  created  disagreements 
indicates  that  the  Guide  is  more  likely  than  the  Gold  Standard  to  rate  High  Duration  jobs 
as  high  risk.  It  also  appears  that  for  jobs  with  multiple  moderate  duration  tasks,  that  the 
Guide  is  more  likely  than  the  Gold  Standard  to  assign  a  medium  rating.  The  Guide 
appears  to  be  more  sensitive  to  task  duration  than  are  the  Gold  Standard  scores. 
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Table  5.4 

Comparison  Between  Beta  Test  Technicians  and  Gold  Standard, 
by  Scenario,  for  the  Overall  Priority  Rating  Scores 


Scenario 

Beta  -  Gold 
Mean  difference 

T-value 

P= 

1 

0.9 

8.53788 

0.00001 

2 

0 

0 

1.00000 

4 

-0.1 

-0.94865 

0.36516 

7 

0.8 

5.691965 

0.00020 

9 

-1.1 

-5.81338 

0.00017 

12 

0 

0 

1.00000 

18 

-0.2 

-0.94867 

0.36515 

19 

-0.4 

-1.42301 

0.18517 

20 

-0.1 

-0.94865 

0.36516 

21 

0.4 

1.716208 

0.11688 

23 

0.9 

3.083582 

0.01157 

24 

-0.2 

-1.42299 

0.18518 

35 

-0.7 

-4.34732 

0.00145 

36 

-0.9 

-4.7564 

0.00077 

38 

-0.6 

-2.57431 

0.02769 

41 

0 

0 

1.00000 

43 

0.8 

5.691965 

0.00020 

45 

0.6 

1.863158 

0.09204 

48 

-1 

-6.36383 

0.00008 

60 

-0.5 

-2.846 

0.01737 

61 

0 

0 

1.00000 

62 

-0.1 

-0.94865 

0.36516 

63 

0.3 

1.333415 

0.21197 

64 

-0.2 

-1.42299 

0.18518 

65 

1 

o 

-2.32375 

0.04250 

66 

-0.1 

-0.94865 

0.36516 

67 

0 

0 

1.00000 

68 

0.8 

5.691965 

0.00020 

69 

-0.4 

-2.32375 

0.04250 

71 

-0.1 

-0.52849 

0.60869 
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Table  5.5 

Comparison  Between  Beta  Test  Technicians  and  Ergonomist  Consensus, 
by  Scenario,  for  the  Overall  Priority  Rating  Scores 


Scenario  Difference  T-value  p= 


1 

-0.1 

-0.948653 

0.36516 

2 

0 

0 

1.00000 

4 

-0.1 

-0.948653 

0.36516 

7 

-0.2 

-1.422991 

0.18518 

9 

-0.1 

-0.528489 

0.60869 

12 

0 

0 

1.00000 

18 

-0.2 

-0.948668 

0.36515 

19 

-0.4 

-1.423008 

0.18517 

20 

-0.1 

-0.948653 

0.36516 

21 

-0.6 

-2.574312 

0.02769 

23 

-1.1 

-3.768823 

0.00367 

24 

-0.2 

-1.422991 

0.18518 

35 

0.3 

1.8631384 

0.09204 

36 

0.1 

0.5284889 

0.60869 

38 

0.4 

1.7162081 

0.11688 

41 

0 

0 

1.00000 

43 

-0.2 

-1.422991 

0.18518 

45 

-0.4 

-1.242105 

0.24254 

48 

0 

0 

1.00000 

60 

0.5 

2.8459959 

0.01737 

61 

0 

0 

1.00000 

62 

-0.1 

-0.948653 

0.36516 

63 

0.3 

1.3334152 

0.21197 

64 

-0.2 

-1.422991 

0.18518 

65 

-0.4 

-2.323745 

0.04250 

66 

-0.1 

-0.948653 

0.36516 

67 

0 

0 

1.00000 

68 

o 

t 

-1.422991 

0.18518 

69 

-0.4 

-2.323745 

0.04250 

71 

-0.1 

-0.528489 

0.60869 

Table  5.6 

Classification  of  Significant  Differences  for 
Overall  Priority  Rating  Scores 


Consistent  Difference-- 

Gold  Difference- 

Split  Difference- 

Expert/Novice  Effect 

Method  Based 

Borderline  Job 

Scenarios 

Scenarios 

Scenarios 

65,69 

1,  7, 9, 36, 38, 43, 68 

21,23, 60 

Table  5.7 

Beta  Test  Job  Ratings  Compared  to 
Gold  Standard  Ratings 


1  Beta  Test  (n=10) 

Low 

Med 

High 

Gold  Low 

4 

3 

3 

Gold  Med 

14 

51 

55 

Gold  High 

7 

56 

117 

5.4  PRACTICALITY 

The  most  significant  aspect  of  the  entire  process  is  the  effectiveness  of  the  Guide  in 
enabling  the  users  to  select  the  appropriate  corrective  actions.  The  measure  of 
practicality  used  in  testing  focused  on  whether  the  Guide  could  be  applied  and  completed 
within  the  time  guidelines  established  by  the  Air  Force. 

5.4.1  Usability.  Information  on  usability  was  obtained  from  technicians  by  the 

ergonomist  who  administered  the  beta  test.  The  beta  test  participants  completed  an 
usability  questionnaire  at  the  end  of  the  beta  test.  The  ergonomist  also  obtained 
comments  throughout  the  testing  and  conducted  a  focus  group/out-briefing  with 
participants  to  obtain  additional  comments  after  the  testing  was  complete. 

The  ergonomist  encouraged  the  technicians  to  ask  clarification  questions  during  the 
testing,  as  needed,  but  pointed  out  that  the  responses  would  be  limited  to  information 
contained  within  the  existing  materials.  In  fact,  most  of  the  techmcians  asked  questions 
whose  answers  were  already  contained  in  the  Job  Scenario  or  in  the  Instructions.  For 
clarification,  the  ergonomist  simply  referred  the  technician  to  that  information.  The 
process  for  responding  to  each  questions  was: 

•  restated  the  question; 

•  directed  the  technician  back  to  the  Instructions  and/or  Job  Scenario;  and 
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•  assisted  the  technician  in  responding  to  the  original  question  in  light  of  the 
new  information. 

The  questions  that  the  technicians  asked  as  they  learned  and  applied  the  Guide  during  the 
testing  were  recorded.  The  content  of  the  questions  provided  information  on  usability. 
The  questions,  responses,  and  impact  on  the  Guide  design  are  provided  in  Table  5.8. 

Table  5.8 

Technician  Questions  and  Experimenter  Responses: 

Lessons  Learned  During  the  Beta  Test 


1  Question 

1  Response 

Question  1.  What  do  RF/DS, 
PEPA  and  EPRA  stand  for? 

Response:  Provided  explanations  of  abbreviations. 

Interpretation:  No  interpretation  necessary. 

Impact  on  the  Final  Guide:  none. 

Question  2.  On  scenario  #18, 
what  is  the  difference 
between  folding/fitting  and 
packing/packaging? 

Response:  Recommended  use  of  best  judgment 

Interpretation:  It  can  be  difficult  to  distinguish  between  tasks 
on  the  video. 

Impact  on  the  Final  Guide:  In  actual  use  a  technician  could 
ask  for  clarification  from  the  person  doing  the  job. 

Question  3.  Should  we  round 
fractions  up  or  down? 

Response:  Referred  participants  to  instructions  and  repreated 
instructions  to  round  up  from  0.5. 

Interpretation:  No  interpretation  necessary. 

Impact  on  the  Final  Guide:  none. 

Question  4.  What  do  we  put 
for  environmental  questions 
if  there  is  no  information 
given  in  the  scenario? 

Response:  Instructed  participants  to  answer  “neutral.” 

Interpretation:  This  is  an  artifact  of  the  testing  situation. 

Impact  on  the  Final  Guide:  none,  data  would  be  available 
when  actually  visiting  a  shop 

The  ergonomist  conducting  the  beta  test  was  also  responsible  for  data  entry.  During  data 
entry,  scoring  mistakes  were  noted.  When  it  was  noticed  that  several  participants  were 
incorrectly  transferring  the  highest  score  instead  of  the  average,  the  ergonomist  corrected 
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the  errors  and  explained  the  correct  method  to  the  participants.  Additional  mathematical 
errors  were  noted  and  corrected  during  data  entry. 

The  usability  questionnaire  consisted  of  13  questions  with  ratings  on  a  five  point  Likert 
scale.  The  questionnaire  is  contained  in  Appendix  C.  One  question  on  the  scale  was 
reversed  (disagree  would  be  the  desired  response  to  “The  tool  is  better  suited  to  an 
ergonomist  than  a  non-ergonomist”).  In  order  to  ease  interpretation,  the  scoring  of  the 
reversed  question  has  been  adjusted  to  match  the  other  questions.  The  average  responses 
for  each  question  fell  between  1.8  and  2.3,  indicating  that  participants  agreed  with  each 
usability  statement.  Only  two  of  the  eleven  subjects  disagreed  with  any  of  the  questions. 
One  of  these  subjects  disagreed  with  6  statements,  while  the  other  subject  only  disagreed 
with  the  statement  “The  risk  criteria  are  easily  estimated  without  measurement.”  The 
similarity  of  responses  for  each  questions  suggests  the  responses  may  reflect  an  overall 
attitude  towards  the  Guide,  rather  than  specific  information  about  individual  questions. 
With  this  in  mind,  it  indicates  that  participants  had  a  generally  favorable  opinion  towards 
the  Guide. 

5.4.2  Time  Requirements.  The  time  requirements  for  completing  the  analysis 
and  for  identifying  and  selecting  control  measures  were  calculated  based  on  the  beta  test 
sessions.  The  mean  time  for  completing  the  Level  I  Checklist  and  scoring  process  was 
16.6  minutes  with  a  standard  deviation  of  10.8  minutes.  The  mean  time  for  identifying 
and  selecting  control  measures  was  12.0  minutes  with  a  standard  deviation  of  7.4 
minutes.  Furthermore  there  was  a  clear  learning  effect,  with  times  decreasing  during  the 
testing.  On  the  first  day  of  testing  the  average  total  time  (checklist  +  summary)  was  59 
minutes.  By  the  fourth  day  performance  time  improved  to  a  total  time  of  23  minutes 
(checklist=13  minutes,  summary=10  minutes).  These  values  are  well  within  the  criteria 
established,  indicating  that  this  Guide  can  be  used  quickly. 

5.4.3  Selection  of  Corrective  Actions. 

5.4  J.l  Beta  Test  vs.  Consensus  Scores:  Corrective  Action  Agreement.  The 

agreement  on  the  selection  of  corrective  actions  was  examined  in  two  directions.  First  the 
percentage  of  time  in  which  a  corrective  actions  which  was  selected  by  a  beta  test  subject 
was  also  selected  by  the  consensus  ergonomist.  Second,  the  piercentage  of  corrective 
actions  selected  by  the  consensus  that  were  also  selected  by  the  beta  testers.  It  was 
determined  that  the  straightforward  agreement  rates  between  the  consensus  ergonomist 
and  beta  testers  would  results  in  an  inflated  agreement  based  on  non-selection  of 
corrective  actions  (since  most  actions  were  not  selected  for  any  given  scenario  agreeing 
not  to  select  a  solution  was  judged  to  be  of  minimal  importance). 

The  143  corrective  actions  were  grouped  according  to  14  basic  types  of  actions  (i.e., 
workstation  modifications,  tool  modifications,  tool  selection).  The  grouping  of  corrective 
actions  is  presented  in  Appendix  D.  The  percentage  of  time  the  beta  test  subjects  selected 
corrective  actions  which  were  the  same  as  the  consensus  was  61%.  This  would  indicate 
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that  out  of  100  corrective  actions  selected  by  the  beta  testers,  61  were  actions  selected  by 
the  consensus  ergonomist. 

The  percentage  of  time  that  a  corrective  action  selected  by  the  consensus  ergonomist  was 
also  selected  by  the  beta  test  subjects  was  70%.  This  indicates  that,  on  average,  whenever 
the  consensus  ergonomist  identified  a  solution  that  7  out  of  10  beta  testers  would  also 
have  identified  that  solution. 

It  appears  that  beta  testers  were  more  likely  to  select  solutions  regardless  of  feasibility. 
The  consensus  ergonomist  appears  more  selective  in  identifying  solutions.  In  actual  use 
this  effect  would  be  minimized  through  the  involvement  of  multiple  people  in  a  single 
implementation. 

5.5  CONCLUSIONS 

The  validation  process  has  assessed  the  reliability,  sensitivity,  validity,  and  practicality  of 
the  Level  I  Ergonomics  Assessment  Guide  for  Maintenance/Inspection  Work  Areas  in 
several  ways,  and  while  there  are  some  issues  that  were  identified  for  improvement  in 
subsequent  Guide  development  efforts,  the  Guide  should  be  considered  to  have 
performed  adequately.  The  specific  detail  summary  is  provided  below. 

For  inter-rater  reliability: 

•  Agreement  rates  between  end  users  were  similar  to  that  which  has  been 
reported  in  the  literature. 

•  For  Overall  Job  Priority  score,  the  process  demonstrated  significant  reliability, 
with  beta  test  subjects  agreeing  55%  of  the  time.  When  comparing  beta  test 
results  to  consensus  and  Gold  Standard  results,  only  2  of  30  jobs  suggested 
differences  between  expert  and  novice  performance.  This  supports  the  notion 
that  the  technicians  obtain  similar  priority  scores  as  experts. 

•  Across  all  questions  in  the  Level  I  Checklist,  the  technicians  agreed  67% 
percent  of  die  time.  This  means  that  technicians  can  identify  the  same  risk 
factors  with  minimal  training. 

•  In  determining  the  selection  of  corrective  action  categories,  the  beta  test 
subjects  selected  the  same  responses  as  experienced  ergonomists  using  the 
Guide  over  60%  of  the  time. 
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For  sensitivity: 


•  There  is  a  significant  relationship  between  the  evaluations  from  the  Air  Force 
technicians  and  the  expert  ergonomists  (both  using  the  same  Level  I  Guide  or 
an  independent  method). 

•  The  Guide  can  significantly  and  correctly  distinguish  between  scenarios  that 
were  judged  by  the  experts  to  be  of  high  or  medium  risk  for  ergonomic  risk 
factor  exposure. 

For  content  validity: 

•  All  of  Job  Factor  questions  are  supported  by  scientific  research.  The  list  of 
Job  Factor  questions  was  comprehensive  in  covering  ergonomic  risk  factors 
which  are  expected  to  occur  in  a  maintenance/inspection  environment.  The 
overall  theoretical  structure  of  the  Guide  is  a  commonly  used  structure  which 
has  been  used  in  other  validated  methods. 

For  concurrent  validity: 

•  When  comparing  beta  test  results  to  consensus  and  Gold  Standard  results, 
only  2  or  30  jobs  suggested  differences  between  expert  and  novice 
performance.  This  supports  the  premise  that  the  technicians  obtain  similar  job 
risk  ratings  as  experts  using  the  Guide.  The  risk  ratings  (priority  scores)  for 
both  technicians  and  experts  using  the  Guide  differed  from  the  gold  standard, 
indicating  that  the  Guide  produces  risk  ratings  that  may  differ  at  times  from 
the  Gold  Standard  ratings.  The  task  duration  appears  to  play  a  larger  role  in 
determining  risk  when  using  the  Guide  than  during  Gold  Standard 
assessments.  This  apparent  bias  may  be  desirable  to  focus  attention  on  the 
most  continually  occurring  tasks. 

For  usability/practicality: 

•  The  Guide  was  easy  to  use  and  can  be  completed  in  well  imder  the  time 
targets  established  by  the  Air  Force.  Usability  comments  provided  by 
technicians  were  favorable. 

•  Technicians  consistently  agreed  in  selecting  corrective  actions.  This  means 
that  technicians  can  be  expected  to  make  the  same  type  of  recommendations  to 
shop  supervisors. 
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•  The  solutions  selected  by  the  technicians  agreed  with  the  solutions  selected  by 
a  consensus  of  ergonomists  61%  of  the  time.  This  result  suggested  that  the 
Guide  assists  technicians  in  generating  solutions  that  experts  would 
recommend.  This  was  one  of  the  primary  objectives  of  the  Guide. 


Opportunities  for  Improvement 

•  The  overall  level  of  agreement  for  specific  questions  selected  for  a  particular 
job  was  lower  than  the  desired  criteria  However,  the  rates  were  comparable 
to  agreement  rates  obtained  in  published  validations  for  other  ergonomics 
assessment  methods.  In  addition,  regardless  of  the  agreement  on  individual 
questions,  this  study  provided  evidence  to  indicate  that  the  Guide  assists 
technicians  in  arriving  at  the  similar  solutions  as  would  be  selected  by  an 
ergonomist. 

•  High  duration  tasks  are  more  likely  to  be  rated  high  risk,  whereas  jobs  with 
multiple,  moderate  diuation  tasks  are  more  likely  to  be  rated  as  medium  risk 
jobs. 


In  sununary,  the  primary  strengths  of  the  Guide  are: 

•  it  is  easy  to  use; 

•  it  has  shown  good  acceptance  by  the  most  likely  end  user  population  (BE 
technicians); 

•  it  meets  and  improves  on  the  “time  for  completion”  requirements  established 
by  the  Air  Force;  and 

•  it  enables  Air  Force  personnel  to  identify  appropriate  corrective  actions  to  the 
ergonomics  risk  factors  identified  in  Maintenance  &  Inspection  jobs  in  a  way 
that  provides  answers  that  are  similar  to  that  which  an  ergonomist  would 
provide,  all  in  a  time-effective  manner. 

The  Level  I  Ergonomics  Assessment  and  Problem-Solving  Guide  for  Work  Areas  can  be 
considered  an  effective  means  for  identifying  and  controlling  ergonomics  hazards  in  work 
areas.  In  addition,  it  represents  a  strong  foimdation  for  developing  similar  approaches  to 
reducing  or  eliminating  the  potential  for  WMDs  in  Warehouse  and  Assembly  Work 
Areas. 
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Biography 


Andrew  J.  Marcotte,  CPE 
Ergonomist 

The  Joyce  Institute/A  Unit  of  Arthur  D.  Little 


Mr.  Marcotte,  MSIE,  CPE,  is  Assistant  Director  for  Ergonomics  for  The  Joyce  Institute,  a 
unit  of  Arthur  D.  Little,  within  the  Environmental,  Health,  and  Safety  Directorate.  He 
specializes  in  industrial  and  office  consulting  and  training,  and  in  the  development  and 
implementation  of  strategic  ergonomics  plans.  He  has  extensive  experience  with  several 
industries  including  the  automotive,  chemical,  metal  processing,  sleep  products  and  office 
product  manufacturing  industries. 

Office  Workplace  Consulting:  Ergonomics  Assessment  and  Design 

•  For  an  international  office  products  company,  Mr.  Marcotte  developed  a  model 
ergonomics  program  plan  designed  to  reflect  the  manufacturing,  s^ety  and 
engineering  management  systems  already  in  place.  The  plan  was  used  to  guide  the 
systematic  implementation  of  ergonomics  design  practices  on  a  “pilot”  basis  at  one  of 
the  company’s  California  facilities.  After  the  plan  was  established,  Mr.  Marcotte 
trained  individuals  (top  management,  engineers,  supervisors,  employee  work  teams) 
who  would  be  responsible  for  implementing  all  aspects  of  the  program. 

(Avery  Dennison,  1994-1995) 

•  For  a  major  U.S.  bedding  manufacturer,  Mr.  Marcotte  developed  and  assisted  in 
implementing  a  cost-effective  abatement  plan  that  addressed  ergonomics  hazards  for 
which  the  company  had  been  cited  and  prepared  the  company  for  responding  to 
OSHA.  Mr.  Marcotte  used  analyses  that  identified  ergonomics  hazards  as  well  as 
production  and  handling  process  inefficiencies,  and  worked  with  the  company 
management  team  to  evaluate  appropriate  handling  technology  and  identified  changes 
to  the  production  scheduling  system.  As  a  result,  the  company  obtained  a  60  percent 
reduction  in  the  OSHA  fine;  reduced  the  number  of  times  each  unit  is  handled  fiom  7 
to  3;  decrease  shipping-related  injuries  fi*om  9  to  1  and  increased  plant-wide 
production  capacity  by  50  percent  at  current  employment  levels  in  a  one  year  period. 
(Adam  Wues^Serta,  1991-1992) 

Industrial  Workplace  Consulting:  Ergonomics  Assessment  and  Design 

•  For  an  aluminum  smelting  and  manufacturing  company,  Mr.  Marcotte  conducted 
advanced  ergonomics  training  for  engineers  and  supervisors  in  smelting,  rolling, 
ingot,  packaging  and  maintenance  areas.  Trained  the  internal  ergonomist  in  advanced 
ergonomics  problem  solving  and  solution  design  techniques  and  certified  the 
ergonomist  to  conduct  in-house  ergonomics  training.  (ALCOA,  1990-1993) 
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•  For  an  iron  foundry,  he  trained  members  of  a  joint  management-union  ergonomics 
committee  to  recognize  the  probable  causes  of  injuries  and  to  devise  prevention 
strategies  based  on  ergonomics  principles.  (Chrysler,  1988) 

•  For  a  metal  fabricator,  Mr.  Marcotte  trained  engineers  and  health/safety  professionals 
in  task  analysis  methodology  to  assist  in  the  identification  of  ergonomic  stressors. 
(Chrysler,  1988) 

•  For  a  paper  products  facility,  Mr.  Marcotte  analyzed  and  recommended  modifications 
to  hand-pacWng  operations  including:  pre-orienting  packages  to  correspond  to  the 
shipper’s  fill  pattern;  providing  guide  rails  to  locate  packages  closer  to  the  operator; 
and  designing  adjustable  height  for  shipper  conveyor  to  allow  for  size  variances. 
(Proctor  &  Gamble,  1989) 

•  He  analyzed  and  recommended  modifications  to  a  materials  handling  operation  for  a 
moist  chemical  product.  Provision  of  a  platform  and  modifications  to  the  material 
container  provided  better  access  and  reduced  bending  and  overhead  reaching. 

(BASF,  1990) 

•  For  an  automobile  and  auto  parts  manufacturer,  he  modified  tool  handling  operations 
in  the  molding  environment  by  adding  roller  tops  to  tooling  transfer  carts.  This 
reduced  operator  fatigue  by  reducing  push/pull  forces  required  to  move  the  tooling 
during  production.  Low  back  stress  was  reduced  and  productivity  increased  by  33%. 
(GM  Central  Foundry,  1986) 

•  For  a  utility/service  industry,  he  identified  equipment  and  tool  design  elements  and 
work  procedures  that  were  contributing  to  back  injuries  for  utilities  service 
technicians.  Recommendations  included  equipment  modifications  as  well  as  safe 
work  practices  and  handling  procedures  for  ladders  used  during  pole  work. 

(CENTEL,  1990-1991) 

•  For  a  medical  instrument  manufacturer,  he  analyzed  and  recommended  modifications 
designed  to  prevent  injuries  on  a  production  line  dedicated  to  wand  assembly  for  bar¬ 
code  data  collection  systems.  Recommendations  targeted  simple  workplace 
modifications  to  improve  the  present  system  as  well  as  product  redesign  elements  to 
improve  manufacturability  and  assembly  of  future  products. 

(Welch  Allyn,  1990-1992) 

•  Mr.  Marcotte  analyzed  sewing  workstations  for  a  major  garment  manufacturer,  and 
recommended  design  modifications  to  provide  adjustable  workstations  based  on  job 
analysis.  This  resulted  in  a  reduction  of  stresses,  pain  and  discomfort  of  the 
operators,  thereby  minimizing  the  potential  for  cumulative  trauma  disorders  in  the 
neck,  shoulders  and  back.  (ILGW,  1988-1989) 
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•  Mr.  Marcotte  trained  engineers  at  a  truck  manufacturing  plant  to  minimize  back 
injuries  and  improve  throughput  by  redesigning  work  area  layout  and  procedures. 
(GM-Moraine  Assembly,  1988-1989) 

•  At  a  meat  processing  plant,  he  evaluated  and  recommended  design  and  layout 
modifications  for  a  pork  processing  line,  to  reduce  reaches  and  grip  forces  and  to 
increase  productivity.  As  a  result,  ergonomics  is  now  being  adopted  into  the  overall 
facilities  and  equipment  design  process.  (Hygrade,  1988-1990) 

Ergonomics  Program  Design,  Management  and  Auditing 

•  For  a  bedding  association,  Mr.  Marcotte  developed  a  strategy  manual  designed  to 
provide  ergonomics  guidelines  that  will  assist  bedding  manufacturers  in  developing 
ergonomics  programs  that  will  reduce  work-related  industries  and  lower  costs.  The 
guidelines  also  are  designed  to  place  the  industry  in  a  strengthened  position  to  ensure 
that,  if  a  federal  ergonomics  standard  proposed  by  the  Occupational  Health  and  Safety 
Administration  (OSHA)  is  adopted,  the  new  guidelines  will  be  reasonable  and 
focused  on  the  industry’s  actual  conditions.  (ISP A,  1 994-1995) 

•  Mr.  Marcotte  developed  guidelines  for  ergonomics  in  the  Chemical  and  Allied 
Industries  through  the  Chemical  Manufacturer's  Association.  (1992) 

Education/Professional  Activities 

Mr.  Marcotte,  a  certified  professional  ergonomist,  earned  his  Master  of  Science  in 
Industrial  and  Operations  Engineering,  specializing  in  Ergonomics/Occupational  Health 
&  Safety  Engineering,  fi’om  the  University  of  Michigan  at  Ann  Arbor.  He  holds  a 
Bachelor  of  Science  in  Industrial  and  Systems  Engineering  fi’om  the  General  Motors 
Institute  at  Flint,  Michigan. 

He  previously  served  on  the  ANSI  -  Z-365  Subcommittee  and  is  a  member  of  the  Human 
Factors  and  Ergonomics  Society,  the  American  Society  of  Safety  Engineers  and  the 
Institute  of  Industrial  Engineers. 

Research/Publications 

Marcotte,  A.  J.,  Barker,  R.  T.  Level  I  Ergonomics  Assessment  Methodology  for 
Maintenance  and  Inspection  Work  Areas,  United  States  Air  Force  Materiel 
Command/Armstrong  Laboratories,  December  1996. 

Marcotte,  A.  J.,  Barker,  R.  T.  Level  I  Ergonomics  Assessment  Methodology  for 
Administrative  Work  Areas,  United  States  Air  Force  Materiel  Conunand/Armstrong 
Laboratories,  October  1996. 

Marcotte,  A.  J.,  Barker,  R.  T.  Job  Requirements/Physical  Demands  Survey  Methodology, 
United  States  Air  Force  Space  Command/Armstrong  Laboratories,  September  1996. 
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"Ergonomics  Applied  to  Product  &  Process  Design  Achieves  Immediate,  Measurable 
Cost  Savings,"  HFES,  San  Diego,  CA,  October,  1995. 

Marcotte,  A.  J.,  Weinkamp,  P.,  Jackson,  P.,  and  Bretz,  T.,  “Moving  Mattresses  Safely:  A 
Case  Study  in  Manual  Materials  Handling,”  International  Ergonomics  Association, 
Warsaw,  Poland,  June  1993. 

Marcotte,  Andrew  and  Adams,  Edie,  “A  Manager’s  Guide  to  Ergonomics  in  the  Chemical 
and  Allied  Industries,”  Chemical  Manufacturers  Association  (CMA),  1992. 

Marcotte,  A.  J.,  “Implementing  a  Successful  Ergonomics  Program  Using  In-House 
Expertise:  Results  Through  Training,  “Proceedings  of  the  lEA  Annual  Meeting,  Paris, 
France,  July,  1991. 

Marcotte,  A.  J.  and  Martino,  John,  “Injury  Prevention  In  An  Incentive-based  Production 
Environment:  A  Case  Study,”  Proceedings  of  the  IFIESR  Annual  Meeting,  Lake  Tahoe, 
Nevada,  June  1991. 

Cook,  Robert  E.  and  Marcotte,  A.  J.,  “Ergonomic  Improvement  in  Games  Manufacturing: 
A  Case  Study,”  Proceedings  of  the  HFS  34th  Annual  Meeting,  Orlando,  Florida,  October 
1990. 

Presentations 

Marcotte,  A.J.,  "Making  a  Business  Case  for  Ergonomics,"  Organization  Resources 
Councelors,  Inc.,  San  Francisco,  CA,  September,  1995. 

Marcotte,  A.J.,  "Ergonomics:  Value-Added  Safety,"  Fisher  Scientific  Conference,  Two 
Rivers,  WI,  March,  1995. 

Marcotte,  A.J.,  "Ergonomics  Workstation  Analysis  for  the  Bedding  Industry," 
International  Sleeps  Products  Association,  Albuquerque,  NM,  March,  1995. 

Marcotte,  A.J.,  "How  to  Develop  a  Practical  Written  Ergonomics  Plan,"  Manufacturers' 
Education  Council,  Columbus,  OH,  February,  1995. 

Marcotte,  A.  J.,  “Ergonomics  Measurable  Results  in  the  Chemical  Industry:  Case 
Studies.”  Presentation  identifying  key  applications  and  results  of  ergonomics 
improvements  in  chemical  process  operations.  National  Safety  Congress,  Chicago,  IL. 
October  1993. 

Marcotte,  A.  J.,  “Public  Utilities  and  Ergonomics:  Problems  and  Solutions.”  Case  study 
presentation  identifying  potential  scope  of  problems  and  results  of  improvements  in 
public  utilities  work.  National  Safety  Congress,  Chicago,  IL.  October  1993. 


Appendix  A 


4 


Biography 


Jeffrey  B.  Nelson,  CPE 
Ergonomist 

The  Joyce  Institute/A  Unit  of  Arthur  D.  Littie 


Mr.  Nelson,  MSIE,  is  a  consultant  for  The  Joyce  Institute/A  Unit  of  Arthur  D.  Little 
specializing  in  industrial  and  office  consulting  and  training  and  in  the  design  of  consumer 
products  for  industrial  and  home  environments.  He  has  extensive  experience  in  office 
product  manufacturing,  automobile  assembly,  semiconductor  and  clean  room 
environments,  and  the  chemical  and  consumer  product  industries. 


Industrial  Workplace  Consulting:  Ergonomics  Assessment  and  Design 

•  For  a  modular  housing  fabricator  in  Mexico,  Mr.  Nelson  designed  a  “super  jig”  for 
use  in  the  construction  of  modular  housing,  and  evaluated  and  critiqued  the  prototype 
of  the  machine. 

•  For  a  major  hospital,  he  evaluated  the  stresses  caused  during  plasma  vial  inspection. 
His  recommendations  included  the  redesign  of  a  new  “reject”  holding  cart,  the 
adjustment  of  a  lighting  screen,  the  relocation  of  task  lighting,  the  implementation  of 
a  worker  rotation  procedure,  and  the  balancing  of  line  speeds. 

•  For  a  corrugated  box  manufacturer,  Mr.  Nelson  identified  the  stressors  placed  on 
workers  involved  in  a  box  assembly  procedure.  He  recommended  process 
improvements  and  equipment  modifications,  including  the  relocation  of  controls, 
scheduled  employee  rotation,  and  the  purchase  of  manual  lifting  devices.  He  also 
recommended  the  purchase  of  automated  folding  equipment,  and  created  a  design  for 
a  stationary  folding  apparatus. 

•  For  a  consumer  products  manufacturer,  he  evalxiated  a  rotational  molding  operation, 
and  reconmiended  process  revisions  as  well  as  design  modifications  for  the  working 
platform,  the  material  delivery  system,  several  tools  and  molds. 

•  For  a  sewage  disposal  facility,  Mr.  Nelson  identified  the  musculoskeletal  stressors 
placed  on  workers  involved  in  the  transfer  of  motor  oil  between  containers.  He 
recommended  the  purchase  of  a  manual  siphon  pump  to  eliminate  static  loading  on 
the  shoulder,  which  was  occurring  during  the  performance  of  several  tasks. 
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•  For  a  snack  foods  manufacturer,  Mr.  Nelson  identified  the  sources  of  musculoskeletal 
stressors  placed  on  workers  involved  in  the  hand  packing  and  box  construction 
processes.  Additionally,  he  was  asked  to  identify  those  benefits  which  would  be 
gained  with  the  implementation  of  further  ergonomics  based  improvements.  Mr. 
Nelson  recommended  the  installation  of  a  mechanized  box  maker  and  label 
applicator,  the  addition  of  a  height-adjustable  table  between  two  conveyor  lines,  and 
the  height  and  weight  constraints  for  the  mylar  packaging  rolls.  He  also 
recommended  modifications  to  the  job  rotation,  and  indicated  the  potential  risk  with 
changes  in  line  speed. 

•  For  the  U.S.  Air  Force-Materiel  and  Space  Conunand,  Mr.  Nelson  served  as  the 
“Gold  Standard  Ergonomist”  during  the  task  evaluation  and  recommendations  phases. 
During  his  lengthy  visits  to  USAF  bases  in  Massachusetts  and  Florida,  he  identified 
and  ranked  those  tasks  which  contained  large  amounts  of  musculoskeletal  stressors. 
Additionally,  for  each  identified  task,  he  devised  engineering-based  solutions  to 
alleviate  those  stressors. 

•  For  the  U.S.  Postal  Service,  Mr.  Nelson  evaluated  the  mechanized  bulk  mail 
sorting/delivery  operation.  He  reconunended  lighting  adjustments  for  all  tasks, 
provision  of  anti-fatigue  matting,  adjustment  of  keypad  features  for  workers  at 
standing  workstations,  creation  of  a  company-wide  job/worker  rotation  policy, 
workstation  redesign  to  facilitate  movement  and  enable  the  use  of  both  hands, 
substitution  of  manual  keypad  entry  with  a  bar-code  reading  system,  and  the  purchase 
of  headsets  and  adjustable  chairs  for  control  room  monitors.  He  also  recommended 
the  incorporation  of  advanced  ergonomics  analysis  techniques  into  future  design. 


Office  Workplace  Consulting:  Ergonomics  Assessment  and  Design 

•  For  a  regional  hospital,  Mr.  Nelson  reconunended  improvements  to  the  Critical  Care, 
Histology  and  Transcription  Departments.  His  recommendations  resulted  in  the 
redesign  of  work  areas  for  enhanced  visibility,  alleviation  of  psychosocial  stresses  and 
increased  desk  space.  He  also  recommended  the  automation  of  certain  activities  to 
reduce  stresses  to  the  upper  limbs  and  remove  external  trauma;  and  recommended  the 
development  of  policy  covering  workload  distribution,  workstation  layout  and 
materials  placement. 

•  For  a  hospital,  Mr.  Nelson  redesigned  a  medical  records  storage  facility.  His 
recommendations  included  the  installation  of  new  equipment  and  the  redesign  of  out¬ 
patient  records  facility. 
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•  For  a  large  technology  firm,  Mr.  Nelson  developed  a  manual  that  instructs  computer 
input  device  users  in  how  to  improve  their  workstation  layout,  body  orientation  and 
work  habits  by  incorporating  ergonomics  principles.  For  the  manual,  he  created  a 
customized  checklist  to  reinforce  these  principles  on  an  ongoing  basis  through  the  use 
of  exercises  and  strategies  to  increase  comfort. 

•  For  a  railway  management  office  where  there  had  been  workplace  injuries,  he 
identified  stressors  and  recommended  improvements  including  the  equipment 
purchase,  institution  of  formal  evaluation  procedures,  redesign  of  work  areas  and 
suggested  design  for  future  space  planning  efforts,  training  in  office  ergonomics. 

•  Mr.  Nelson  evaluated  physical  and  psychosocial  issues  involved  with  an  aerospace- 
industry  office  worker  who  had  been  diagnosed  Fibromyalgia.  His  recommendations 
included  workstation  redesign  and  enhancements,  education  about  Fibromyalgia, 
broadening  of  the  worker’s  job  to  include  greater  responsibility  and  variety  of  tasks, 
and  training  in  office  ergonomics. 


Product  Design  Criteria  Development  for  End-User 

•  For  a  multinational  computer  design  firm,  Mr.  Nelson  developed  design  and 
operational  criteria  to  be  used  in  the  development  of  a  computer  input  device.  The 
criteria  will  allow  for  movement/control  in  all  planes  and  force  feedback  without 
causing  potential  injury. 


Training  Design  and  Implementation 

•  For  the  U.S.  Postal  Service,  Mr.  Nelson  developed  and  implemented  customized 
ergonomics  course  material,  addressing  those  ergonomics  stressors  for  tasks 
involving  sitting,  standing  and  materials  handling. 

•  Mr.  Nelson  has  trained  employees  in  ways  to  identify,  alleviate  and  prevent 
workplace  ergonomics  hazards  at  numerous  industrial  and  office  firms,  including  a 
chemical  manufacturer,  an  electrical  component/semiconductor  manufacturer,  an 
electronic  test  equipment  manufacturer,  a  hospital  equipment  manufacturer, 
newspaper  printing  and  publishing  companies,  manufacturers  of  office,  vinyl  and 
paper  products,  an  international  petrochemical  company,  a  major  pharmaceuticals 
firm,  a  pulp  and  paper  processor,  a  seafood  processor,  a  manufacturer  of  silicon 
wafers,  a  truck  manufacturer  and  a  wood  and  steel  cabinet  construction  firm. 
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Education/Professional  Activities 

Mr.  Nelson,  a  Certified  Professional  Ergonomist,  received  a  Master  of  Science  in 
Industrial  Engineering,  specializing  in  Industrial  Ergonomics,  from  the  University  of 
Cincinnati  in  Ohio.  He  holds  a  Bachelor  of  Science  in  Kinesiology,  specializing  in 
Biomechanics,  Neurology  and  Physiology  from  the  University  of  California  at  Los 
Angeles. 

He  is  a  member  of  the  Human  Factors  and  Ergonomics  Society  and  the  American  Society 
of  Safety  Engineers. 

Publications 

Nelson,  Jeffrey  B.  and  Anil  Mital,  “An  Ergonomical  Evaluation  of  the  Primary  Hand 
Flexibility  and  Capability  Changes  with  Increases  in  Examination/Surgical  Glove 
Thickness,”  Ergonomics,  Vol.  38,  No.  4,  April  1995. 

Presentations 

Nelson,  J.  B.  and  Joyce,  M.  “Measuring  the  Results  of  Ergonomics  Training,”  ErgoCon, 
San  Jose  CA,  May  1995. 

Nelson,  J.  B.,  and  Joyce,  M.  “Computer  Related  Injuries:  Ergonomics  and  OSHA 
Guidelines,”  Society  of  Architectural  Administrators,  Los  Angeles  CA,  May  1994. 

Nelson,  J.  B.  and  Joyce,  M.  “Ergonomics,”  Western  Safety  Congress,  Anaheim  CA,  May 
1994. 
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Table  B 

Alpha  Test  Results  Suggested  the  Following  Changes  to  Enhance  Usability 


1  Concern 

Modification 

Low  task  duration  category  under-rated  infrequent 
lifting  tasks  and  contributed  to  overall  uncertainty 
in  selecting  categories 

Eliminated  low  duration  category.  All  lifting  treated 
as  at  least  moderate  duration.  Weighting  factors  for 
lifting  related  job  factors  increased. 

Methodology  appeared  to  over-rate  the 
shoulder/neck  and  hands/wrists  body  regions 

The  weighting  for  some  job  factors  within  these 
body  areas  were  reduced.  Job  factor  questions 
within  these  body  areas  were  combined  or 
eliminated  to  reduce  total  number  of  possible  points 
for  a  body  area. 

In  some  cases,  raters  forgot  to  carry  subtotal  across 
to  the  second  page  when  hands/wrists/arms  body 
region  job  factor  questions  extended  across  two 
pages. 

The  reduced  number  of  questions  allowed 
hand/wrist/arm  to  fit  on  one  page,  eliminating  this 
potential  problem. 

Alpha  test  subjects  were  concerned  that  the  two 
separate  and  force  questions  (one  for  static  and  one 
for  repeated)  lowered  agreement  on  the  alpha  test 
and  would  create  greater  amounts  of  disagreement 
among  novice  users. 

The  two  questions  were  combined  into  a  single 
question. 

Alpha  test  subjects  were  concerned  that  the  wording 
of  the  job  factor  questions  for  the  legs/knees/feet 
were  somewhat  ambiguous,  causing  experienced 
ergonomists  to  disagree  on  correct  responses  to  the 
same  identified  hazards. 

The  wording  of  the  job  factor  questions  for 
legs/knees/feet  were  modified  to  improve  clarity. 

Alpha  test  subjects  were  concerned  with  the  clarity 
of  wording  on  a  number  of  individual  questions. 

Minor  modifications  to  the  wording,  examples  and 
criteria  (>10  degrees)  were  incorporated  to  improve 
clarity. 
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Usability  Questionnaire 
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APPENDIX  D 


Grouping  of  Corrective  Actions  to  Determine  Corrective  Action  Agreement 
Between  Beta  Testers  and  Concensus  Ergonomists 
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Grouping  of  Corrective  Actions  to  Determine  Corrective  Action  Agreement 
Between  Beta  Testers  and  Concensus  Ergonomists 


1  Groupings 

Corrective  Action  Numbers 

1.  Design 

16,  37, 99, 101,  102, 128, 129, 140 

2.  Manual  Material 

4, 26, 36, 46,  58,  61,  69,  71, 78,  83,  106, 107,  119,  126,  127,  130, 

Handling 

131,139, 142 

3.  Workstation 

9, 24, 31, 32,  38, 41,  57,  63, 64,  79,  80,  81,  82,  90,  1 12,  1 14,  1 15, 
117,118, 123, 124, 132, 136 

4a.  Tool  Modification 

3, 21, 33, 34,  54,  68,  72,  74,  88, 91, 92, 94, 103, 121, 133,  137 

4b.  Different  Tool 

10,  29, 43,  55,  59, 62, 66,  70, 75,  76,  77,  89, 98 

5.  Work  Practices 

i,  2,  5,  8, 11,  12, 13, 20, 25, 42, 97,  138, 141 

6.  Visual 

14, 18, 22, 27,  30, 60,  84, 100,  108,  109 

7.  Temperature 

7,  15,23,93,104, 105,110 

8.  Cart/Floor 

17,  19, 35, 47, 48, 67 

9.  Office 

39, 40, 44, 45, 49, 53,  56,  65,  73,  85, 1 1 1,  122 

10.  Tool  Support 

116 

11.  Other 

113 

12.  Armstrong  Labs 

6 

13.  Chair 

28,  87,  120 

14.  Foot 

50,  51,  52,  86, 95, 96,  134, 135, 143 
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APPENDIX  E 


Raw  Data:  Alpha  and  Beta  Test 


This  page  intentionally  left  blank 


Appendix  E 


ALPHA  TEST  SUMMARY 


Appendix  E 


This  page  intentionally  left  blank 


Appendix  E 


Appendix  E  Alpha  Test 


Alpha  Test  S 


S' 


(0 


Appendix  E  Alpha  Test  - 


Appendix  E  Alpha  Test  - 


Appendix  E  Alpha  Test  - 


Appendix  E  Alpha  Test  - 


Appendix  E  Alpha  Test  - 


Appendix  E  Alpha  Test  - 


Appendix  E  Alpha  Test  - 


It 


1^ 


raa 


COlCsl  |CO|CO|COjCM|<M|CO|C9  CM 


■■■ 


Appendix  E  Alpha  Test  - 


Appendix  E  Alpha  Test  - 1 0 


Appendix  E  Alpha  Test  -11 


Alpha  Test  Summary 


Appendix  E  Alpha  Test  - 1 2 


Appendix  E  Alpha  Test  - 13 


B 


P 

I 

5J 

Iff' 


?  =>  r 


■■HHI 


Appendix  E  Alpha  Test  - 15 


Appendix  E  Alpha  Test  - 1 6 


III 

imnni 

in 

■  n 

■■■■■■■ 

1  ■ 

n 

■■ 

nil 

■HH 

in 

■  n 

nil 

■m 

§n 

1 

n 

■  n 

mi 

■■■I 

in 

1 

n 

tn 

mi 

■■■ 

in 

1 

n 

mu 

mi 

1  ■ 

1 

n 

■  11 

nil 

mu 

n 

im 

■  n 

nil 

mu 

1  Ul 

1 

un 

nil 

■■ 

Ul 

in 

un 

nil 

■■ 

I  III 

un 

nil 

nil 

■  n 

nil 

llll 

■  n 

nil 

in 

■  n 

nil 

■■ 

n 

in 

111 

nil 

mm 

n 

inni 

mini 

nil 

imn 

Ul 

imn 

mini 

nil 

imn 

iinm 

mim 

linn 

mm 

nin 

mm 

S  is  IS  IS  IS  IS  15^  IS  I?  15 1?  I?  IS  IS  IS  IS 


Appendix  E  Alpha  Test  - 17 


Appendix  E  Alpha  Test  - 1 8 


Appendix  E  Alpha  Test -19 


Alpha  Test  Summary 


Appendix  E  Alpha  Test  -  20 


Appendix  E  Alpha  Test  -21 


Appendix  E  Alpha  Test  -  22 


Appendix  E  Alpha  Test  -  23 


Appendix  E  Alpha  Test  -  24 


Alpha  Test  Summary . 


Appendix  E  Alpha  Test  -  25 


Appendix  E  Alpha  Test  -  26 


Appendix  E  Alpha  Test  -  27 


Appendix  E  Alpha  Test  -  28 


Appendix  E  Alpha  Test  -  29 


Appendix  E  Alpha  Test  -  30 


Appendix  E  Alpha  Test  -  3 1 


Appendix  E  Alpha  Test  -  32 


Alpha  Test  Summary 


Appendix  E  Alpha  Test  -  33 


Appendix  E  Alpha  Test  -  34 


Appendix  E  Alpha  Test  -  35 


Appendix  E  Alpha  Test  -  36 


Appendix  E  Alpha  Test  -  37 


Appendix  E  Alpha  Test  -  38 


Appendix  E  Alpha  Test  -  39 


Appendix  E  Alpha  Test  -  40 


Appendix  E  Alpha  Test  -  41 


Appendix  E  Alpha  Test  -  42 


1  nil 

■■ 

imm 

mamim 

■■ 

IHIH 

maam 

■■ 

!■■■ 

maaam 

■■ 

imm 

'oaaam 

■■ 

IHIB 

'maamn 

wm 

IBIB 

^maama 

■■ 

IBiB 

imm 

IBIB 

mm 

IBIB 

lam 

II 

lam 

II 

'mm 

mam 

imm 

wmm 

lam 

■H 

tmm 

■■■ 

II 

'mm 

■■■ 

Appendix  E  Alpha  Test  -  43 


Appendix  E  Alpha  Test  -  44 


I 

i; 


1st 


i 


■  ■■■i 

1  r 

I 

■  I 


II 
II 
II 

ji 
j  ■ 

■  ■■■■ 


Appendix  E  Alpha  Test  -  45 


Alpha  Test  Si 


I- 


Appendix  E  Alpha  Test  -  47 


Van 


BETA  TEST  SUMMARY: 
OVERALL  AND  CORRECTIVE  ACTIONS 


Appendix  E 


This  page  intentionally  left  blank 


Appendix  E 


Appendix  E  Beta  Test  - 


Beta  Test  Summary 


Appendix  E  Beta  Test  - 


Beta  Test  Summary 


Appendix  E  Beta  Test  - 


Appendix  E  Beta  Test  - 


Beta  Test  Summary 


Appendix  E  Beta  Test  - 


Beta  Test  Summary 


Appendix  E  Beta  Test  - 


Analyst  Scenario  ckisttime  Sum  Time  Order  25  26  27  28  29  30  SN  Score  HWA  Score  BT  Score  LF  Score  HE  Score  |SN  Rating  |HWA  Rating 

BetaH  69  25  17  2  0  0  0  0  0  0  4  2  6  5  0  2 


Appendix  E  Beta  Test  - 


ffliiiiiiiiiiniliiniiliiiii 


nsuBinBUSi^isiiim 


Appendix  E  Beta  Test  - 


Appendix  E  Beta  Test  - 


Appendix  E  Beta  Test  - 1 0 


MBM—li 


04\06 


LF  Rating  |HE  Rating  |Job  Rating  |CA1  [CA2  |CA3  |CA4  |CAS  |CA6  |CA7  |CA8  |CA9  |CA10  |CA11  |CA12  |CA13  |CA14  |CA1S  |CA16 


|r-|CO!cOiCM|eM|CgtCO|<>4  CMCO  T~ 


■■i 


CO 

lO 

CJ 

CM 

tu 

LU 

LiJ 

UJ 

111 

ID 

ID 

ID  ID 

LD 

ID 

ID 

« 

is 

w 

« 

is 

« 

©  « 

© 

© 

© 

0) 

© 

0) 

© 

0) 

©  © 

© 

© 

© 

m 

CQ 

CQ 

CD 

CD 

CD 

CQ 

CQ  CQ 

CQ 

CO 

CQ 

Appendix  E  Beta  Test  - 12 


04fMco<ocoeMeocMfocoeococM 


Appendix  E  Beta  Test  - 13 


Scenario  |BT  Rating  |LF  Rating  |HE  Rating  |Job  Rating  |CA1  |CA2  |CA3  |CA4  |CAS  |CA6  |CA7  |CA8  CA9  CA10  CA11  CA12  CA13  CA14  CA15  |CA16 


Appendix  E  Beta  Test  - 14 


Analyst  Scenario  BT  Rating  LF  Rating  HE  Rating  Job  Rating  CA1  CA2  CA3  CA4  CAS  |CA6  |CA7  |CA8  |CA9  |CA10  |CA11  |CA12  |CA13  |CA14  |CA1S  |CA16 
BetaH  69  2  2  1  2  T - 


Appendix  E  Beta  Test  - 1 5 


Appendix  E  Beta  Test  - 1 6 


CA17  CA18  CA19  CA20  CA21  CA22  CA23  CA24  CA25  CA26  CA27  CA28  CA29  CA30  CA31  CA32  CA33  CA34  |CA35  |CA36  CA37 


Appendix  E  Beta  Test -17 


CA18  ICA19  ICA20  |CA21  ICA22  ICA23  ICA24  ICA25  |CA26  |CA27  |CA28  |CA29  |CA30  |CA31  |CA32  |CA33  |CA34  |CA3S  |CA36  |CA37 


Appendix  E  Beta  Test  -  1 8 


CA21  ICA22  ICA23  |CA24  |CA25  |CA26  |CA27  |CA28  |CA29  |CA30  |CA31  |CA32  |CA33  |CA34  ICA35  |CA36  |CA37 


II 

II 

I 


Appendix  E  Beta  Test  - 19 


CA20  ICA21  ICA22  |CA23  {CA24  |CA25  |CA26  |CA27  |CA28  |CA29  |CA30  |CA31  {CA32  |CA33  |CA34  |CA35  |CA36  |CA37 


|H 

iHI 


Appendix  E  Beta  Test  -  20 


Appendix  E  Beta  Test  -  2 1 


Appendix  E  Beta  Test  -  22 


Analyst  |Scenario  |CA17  |CA18  |CA19  CA20  CA21  CA22  CA23  CA24  CA25  CA26  CA27  CA28  CA29  CA30  CA31  CA32  CA33  CA34  CA3S  CA36  CA37 
BetaH  M  1  _ 


Appendix  E  Beta  Test  -  23 


Appendix  E  Beta  Test  -  24 


Appendix  E  Beta  Test  -  25 


Appendix  E  Beta  Test  -  26 


Appendix  E  Beta  Test  -  27 


Appendix  E  Beta  Test  -  28 


Appendix  E  Beta  Test  -  29 


Analyst  |Scenario  |CA38  |CA39  CA40  CA41  CA42  CA43  CA44  CA4S  CA46  CA47  CA48  CA49  CASO  CAS1  CAS2  CA53  CA54  CA55  CA56  CAST  CA58 
BetaG  M  i"  1  1 


Appendix  E  Beta  Test  -  30 


Appendix  E  Beta  Test  -31 


CA40  |CA41  |CA42  |CA43  |CA44  |CA4S  |CA46  |CA47  |CA48  |CA49  |CASO  |CAS1  |CAS2  |CAS3  |CAS4 TCASS  ICAS6  ICAS7  ICAS8 


Appendix  E  Beta  Test  -  32 


Appendix  E  Beta  Test  -  33 


Appendix  E  Beta  Test  -  34 


CAS9  ICA60  ICA61  |CA62  |CA63  ICA64  |CA6S  |CA66  |CA67  |CA68  |CA69  |CA70  |CA71  |CA72  |CA73  ICA74  |CA75  |CA76  |CA77  ICA78  ICA79 


Appendix  E  Beta  Test  -  35 


CASS  ICA60  |CA61  |CA62  |CA63  |CA64  |CA6S  |CA66  |CA67  |CA6S  |CA69  |CA70  |CA71  |CA72  |CA73  |CA74  |CA75  |CA76 


n 

II 

1 

II 

II 

iHiiinni 

Appendix  E  Beta  Test  -  36 


Appendix  E  Beta  Test  -  37 


Analyst  |Sc«nario  |CA59  CA60  CA61  CA62  CA63  CA64  CA65  CA66  CA67  CA68  CA69  CATO  CA71  CA72  CA73  CA74  CATS  CA76  CA77  CA78  CA79 

BetaG  I  6o1  l1  |  |  i"  1  1  I  1  |  Tl  i1 


■■■1 


I 


Appendix  E  Beta  Test  -  38 


Appendix  E  Beta  Test  -  39 


Appendix  E  Test  -  40 


Appendix  E  Beta  Test  -  41 


Appendix  E  Beta  Test  -  42 


Scenario  |CA80  |CA81  |CA82  |CA83  |CA84  |CA85  |CA86  |CA87  |CA88  |CA89  |CA90  ICA91  |CA92  |CA93  ICA94  |CA95  |CA96  |CA97  |CA98  |CA99  ICAIOO 


Appendix  E  Beta  Test  -  43 


CA80  ICABI  ICA82  |CA83  |CA84  |CA85  |CA88  |CA87  |CA88  1CA89  |CA90  ICASI  1CA92  ICASS  1CA94  |CA95  |CA96  |CA97  ICASS  |CA99  ICAIOO 


Appendix  E  Beta  Test  -  44 


Appendix  E  Beta  Test  -  45 


Appendix  E  Beta  Test  -  46 


Appendix  E  Beta  Test  -  47 


Scenario  ']CA80  |CA81  |CA82  |CA83  |CA84  ICASS  |CA86  |CA87  |CA88  |CA89  |CA90  |CA91  |CA92  |CA93  |CA94  |CA9S  |CA96  |CA97  |CA98  |CA»9  ICAIOO 


Appendix  E  Beta  Test  -  48 


Appendix  E  Beta  Test  -  49 


CA102  ICA103  ICA104  CA10S  |CA106  |CA107  |CA108  |CA109  ICAIIO  ICAIII  |CA112  |CA113  ICA114  ICAIIS  ICA116  ICA117  ICA118 


Appendix  E  Beta  Test  -  50 


Appendix  E  Beta  Test  -  5 1 


ix  E  Beta  Test  -  52 


Appendix  E  Beta  Test  -  53 


Appendix  E  Beta  Test  -  54 


Scenario  [CAIOI  |CA102  |CA103  [CA104  |CA10S  |CA106  |CA107  |CA108  |CA109  |CA110  |CA111  |CA112  |CA113  |CA114  |CA115  |CA116  |CA117 


Appendix  E  Beta  Test  -  55 


Appendix  E  Beta  Test  -  56 


Appendix  E  Beta  Test  -  57 


Appendix  E  Beta  Test  -  58 


CA120  ICA121  ICA122  |CA123  |CA124  |CA126  |CA127  ICA128  |CA129  |CA130  |CA131  |CA132  |CA133  ICA134  |CA13S  |CA136  ICA137 


Appendix  E  Beta  Test  -  59 


CA120  |CA121  |CA122  |CA123  |CA124  |CA126  |CA127  |CA128  |CA129  ICAISO  |CA131  |CA132  |CA133  |CA134  |CA135  ICA136  |CA137 


Appendix  E  Beta  Test  -  60 


Appendix  E  Beta  Test  -  6 1 


Appendix  E  Beta  Test  -  62 


Scenario  |CA119  |CA120  |CA121  |CA122  ICA123  |CA124  |CA126  ICA127  ICA128  |CA129  ICAISO  ICA131  |CA132  |CA133  |CA134  ICA13S  |CA136  |CA137 


Appendix  E  Beta  Test  -  63 


Appendix  E  Beta  Test  -  64 


§• 


p 

xn 

c/a 

O 

H 

<u 

ffl 


Appendix  E  Beta  Test  -  66 


Beta  Test  Summary 


CA143  1 

CA142 

CA141 

o 

< 

U 

CA139 

CA138 

Scenario 

|€9 

s 

|S9 

|99 

(O 

l89 

69 

04\06 

07\08 

12\33 

24\25 

CM 

i 

o> 

CO 

C3> 

o 

CM 

CM 

CO 

CM 

9C 

<o 

CO 

oo 

CO 

CO 

! 

45 

CO 

-M* 

09 

(O 

CM 

CO 

£9 

s 

CO 

99 

CO 

89  ! 

69  1 

104\06 

jAnalyst 

iBetaC 

o 

« 

© 

CO 

[BetaC  1 

|BetaC  1 

iBetaC 

IBetaC 

1  BetaC 

1  BetaC 

1  BetaC 

iS 

CD 

<0 

o 

CD 

IBetaC 

|BetaD 

Q 

(0 

© 

CD 

|BetaD 

Q 

© 

CD 

|BetaD 

Q 
21 
©  1 
CO  ! 

a 

2 

© 

CD 

iBetaD 

D 

m 

© 

CD 

|BetaD 

Q 

ra 

© 

CD 

O 

© 

© 

CD 

|BetaD 

o 

2 

”© 

|BetaD 

IBetaD 

|BetaD 

Q 

2 

© 

CD 

Q 

2 

© 

m 

I 

CO 

|BetaD 

IBetaD 

|BetaD 

IBetaD 

|BetaD 

o 

2 

© 

m 

|BetaD 

Appendix  E  Beta  Test  -  67 


Scenario  ICA138  |CA139  |CA140  |CA141  |CA142 


I- 


s 


H 

V 

CQ 


Appendix  E  Beta  Test  -  71 


Beta  Test  S' 


& 


fO 

< 

O 

CA142 

▼-* 

CA141  1 

CA140 

CA139 

CA138 

Scenario 

00 

o 

CN 

T— 

CM 

CO 

CM 

to 

CO 

36 

38 

CO 

■M- 

to 

00 

09 

Jo 

CM 

(O 

eg 

CO 

99 

67 

89 

69  I 

|04\06 

|07\08 

|12\33 

to 

CM 

[Analyst 

[BetaJ 

[BetaJ  I 

[BetaJ 

[BetaJ 

[BetaJ 

[BetaJ 

[BetaJ 

[BetaJ 

w 

a5 

CD 

[BetaJ 

[BetaJ 

I  BetaJ 

[BetaJ 

[BetaJ 

[BetaJ 

(0 

a> 

m 

-> 

iS 

OJ 

m 

[BetaJ 

[BetaJ 

“O 

iS 

00 

[BetaJ 

[BetaJ 

[BetaJ 

[BetaJ 

[BetaJ 

CD 

CD_ 

[BetaJ 

Appendix  E  Beta  Test  -  72 


BETA  TEST  BY  SUBJECT 


Appendix  E 


This  page  intentionally  left  blank 


Appendix  E 


Beta  Subject  A 


r 


tf) 

CM 

u> 

“cvT 

gggggggggggggggggggggggggggygggggg 

:  QQQggggggggggggmimBgggggggggCjmgggiil 


gMm|CTrararararagg^^|3CTgBgMMMBjgjgBBB||OBBraOB 

HgggggggBggggggggggggggggggggggggg 

HHHHHHHHHHHiilMMmMmmMMIMl 

HHHlHliiMimiMMmMlMMMMMl 

1 


HMHHMMH— mmiBiiimmmmi 

gagggBggggHHHHUHMimmimimm 


€0 


Appendix  E  Beta  Subject  A  - 


ggggggggggggggmgggggBggg 

-gBggggHjg^^mgg^gggg 


C>J|<M|<M|CO|CO|CMl^lO  «M  CM  CM 


gggmmgggggggmg^ggg 

mw 


hT' 


ggggggggggggggggggggggggg 

ggggggggggggggmgggggggg 


HHHHHHHHHHHHHHHHHHHHHHHHH 


<O|C0|CO|O4|<D|<O!COleOlCOj'r-|C>Jjf0  e0 


HHHmHHHHHMMHiMiiiilM 

■lUlWMilllMlliliW 

ggggggggggggggggggggggggg 

mm— WMMWMMl 


I 

ca 

«o 

mi 

o 

Hq 

Appendix  E  Beta  Subject  A  - 


Appendix  E  Beta  Subject  A  - 


Appendix  E  Beta  Subject  A  - 


Beta  Subject  B 


Appendix  E  Beta  Subject  B  - 


Beta  Subject  B 


HHHIlHnHHHIlH 


Appendix  E  Beta  Subject  B  - 


Beta  Subject  B 


Appendix  E  Beta  Subject  B 


Beta  Subject  B 


Appendix  E  Beta  Subject  B  - 


Beta  Subject  C 


Appendix  E  Beta  Subject  C  - 


Appendix  E  Beta  Subject  C  - 


Beta  Subject  C 


Appendix  E  Beta  Subject  C  - 


Appendix  E  Beta  Subject  C  - 


Beta  Subject  D 


Appendix  E  Beta  Subject  D  - 


Beta  Subject  D 


Appendix  E  Beta  Subject  D  - 


Beta  Subject  D 


Appendix  E  Beta  Subject  D  - 


Beta  Subject  D 


Appendix  E  Beta  Subject  D  - 


Beta  Subject  E 


Appendix  E  Beta  Subject  E  - 


Beta  Subject  E 


Appendix  E  Beta  Subject  E  - 


Appendix  E  Beta  Subject  E  - 


Appendix  E  Beta  Subject  E  - 


Beta  Subject  F 


Appendix  E  Beta  Subject  F  - 


Beta  Subject  F 


Appendix  E  Beta  Subject  F  - 


Beta  Subject  F 


Appendix  E  Beta  Subject  F  - 


Beta  Subject  F 


Appendix  E  Beta  Subject  F  - 


Beta  Subject  G 


Appendix  E  Beta  Subject  G  - 


Beta  Subject  G 


CM 

to 

CO 

CO 

1 

CO 

CM 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

CM 

1 

1 

1 

1 

1 

1 

1 

1 

r 

1 

1 

1 

o 

1 

1 

1 

CM 

CM 

CM 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

o 

1 

1— 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

o 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

r 

CM 

CO 

1 

1 

1 

1 

CM 

CM 

CM 

CM 

1 

1 

1 

CM 

CO 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

O 

1 

o 

1 

o 

o 

1 

1 

1 

1 

1 

1 

1 

o 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

O 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

g] 

El 

1 

1 

1 

I 

1 

1 

(O 

o 

1 

CM 

CM 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

o 

1 

o 

CO 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

■ 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

■ 

1 

1 

1 

1 

CM 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

o 

1 

1 

1 

1 

1 

1 

1 

o 

CM 

1 

1 

1 

1 

1 

1 

1 

O 

to 

1 

1 

1 

1 

CO 

1 

1 

1 

1 

o 

I 

1 

1 

1 

1 

1 

1 

CO 

To 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

CO 

o 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

CO 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

■ 

1 

1 

1 

1 

1 

i 

I 

1 

CM 

1 

1 

1 

1 

1 

1 

1 

1 

CM 

CM 

1 

1 

i 

1 

1 

CM 

CM 

CM 

to 

1 

1 

1 

1 

1 

1 

1 

1 

CO 

CO 

CO 

1 

1 

1 

1 

1 

1 

I 

CO 

CO 

1 

I 

1 

1 

1 

1 

1 

1 

o 

1 

1 

■ 

1 

1 

1 

1 

1 

1 

1 

o 

1 

1 

1 

1 

1 

1 

1 

to 

to 

1 

CM 

CM 

1 

1 

1 

1 

1 

1 

1 

CM 

CM 

1 

1 

1 

1 

CO 

1 

1 

1 

1 

1 

CO 

1 

1 

1 

1 

1 

1 

1 

1 

CM 

II 

1 

1 

1 

1 

CO 

CO 

i 

1 

■ 

1 

1 

1 

1 

i 

i 

I 

1 

CM 

1 

1 

CO 

CD 

CM 

CO 

o 

h- 

CO] 

to 

C3> 

■ 

■ 

CM 

CM 

CM 

CO 

o 

CM 

CO 

00 

SI 

SI 

CM 

1 

1 

1 

r— 

1 

1 

1 

1 

I 

■ 

1 

1 

Cl 

■s 

CM 

CO 

CO 

CO 

rn 

(O 

00 

Oi 

T- 

T* 

(O 

fD 

CO 

CD 

CO 

CO 

CO 

1 

CO 

CO 

CO 

CO 

Appendix  E  Beta  Subject  G  - 


Appendix  E  Beta  Subject  G  - 


Beta  Subject  G 


1 

1 

1 

1 

1 

CM 

T- 

1 

CM 

CN 

1 

1 

1 

1 

CO 

CM 

CM 

CM 

1 

1 

1 

1 

1 

ym. 

1 

1 

1 

CM 

CM 

1 

1 

1 

1 

CO 

CO 

CM 

CM 

r 

CO 

CO 

CO 

CO 

CM 

CM 

CO 

CM 

CM 

CO 

CO 

CM 

1 

o' 

o' 

1 

CM 

CM 

1 

o 

o 

o 

1 

1 

1 

CO 

CM 

CO 

o 

CO 

_ 

1 

1 

1 

lO 

IS. 

1 

o 

1 

CM 

CO 

r 

CM 

tn 

o 

CO 

to 

lO 

CO 

CM 

oo 

CO 

CO 

& 

CD 

CD 

1 

PI 

1 

1 

CM 

(O 

CO 

CO 

CO 

1 

to 

CO 

lO 

CO 

99  1 

99  1 

99  1 

99  1 

CO 

CO 

IS- 

CO 

CO 

CO 

69  1 

Appendix  E  Beta  Subject  G  - 


Appendix  E  Beta  Subject  H  - 


Beta  Subject  H 


Appendix  E  Beta  Subject  H  - 


Appendix  E  Beta  Subject  H  - 


Beta  Subject  H 


Appendix  E  Beta  Subject  H  - 


Beta  Subject  I 


CO 

CO 

CM 

t^mm 

innn 

CM  CM  -f-  O  O  O 


O  O  O  O 


■ 


IHIHII 


I 


^  I  CO  I  CO  I  CO  fO  CO 


CO 

o 

CO 

Appendix  E  Beta  Subject  I 


Appendix  E  Beta  Subject  I  - 


Appendix  E  Beta  Subject  I  - 


Appendix  E  Beta  Subject  I  - 


Beta  Subject  J 


Appendix  E  Beta  Subject  J  - 


Appendix  E  Beta  Subject  J  - 


Appendix  E  Beta  Subject  J  - 


Appendix  E  Beta  Subject  J  - 


APPENDIX  F 

Alpha  and  Beta  Test  Checklist  and  Scoring  Summaries 
(Draft  1,  Draft  2  and  Draft  3) 


L 


This  page  intentionally  left  blank 


Appendix  F 


DRAFT  1 


Appendix  F 


This  page  intentionally  left  blank 


Appendix  F 


USAF  Maintenance  Tool 

Level  I  -  Ergonomics  Assessment  for  Maintenance  and  Inspection  Work  Areas  Page  1 


Parti  -  Work  Content  (Description  of  Tasks  Performed) 


For  this  section,  work  with  the  employee  to  obtain  a  basic  description  of  the  types  of  tasks  that  make  up  the  job  being 
evaluated.  Using  the  Maintenance  and  Inspection  Work  Content  (description  of  tasks  performed  list),  write  in  the  tasks  that 
the  employee  performs  and  indicate  the  approximate  task  frequency.  If  the  employee  mentions  a  task  which  is  not  currently 
included  on  the  task  list,  write  the  additional  task  and  work  frequency. 


•  Routine: 

—  >  50  %  (BGlgh):  The  total  percentage  the  job  spent  performing  the  task  is 
greater  than  50%. 

—  10-50  %  (Moderate):  The  total  percentage  of  the  job  spent  performing  the 
task  is  between  10  and  50%. 

•  <  10  %  (Low):  The  total  percentage  of  the  job  spent  performing  the  task  is  less 
than  10%. 

•  Non-routine:  The  task  is  not  performed  during  the  normal  course  of  work  or  is 
performed  under  special  circumstances. 


WORK  CONTENT  MATRK 


Task 

Lifting  / 

Exertion 
occur  in  Task 

(Chec 

kone) 

Routine  I 

Non-Routine 

(Low) 

(Moderate) 

(High) 

<  10% 

10-50%. 

>50% 

1. 

O 

o 

o 

o 

o 

2. 

o 
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3. 

o 

o 

o 

o 
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4. 
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o 

o 

5. 

o 

o 

o 

o 

o 

6. 

o 

o 

o 

o 

o 

Critical  tasks  include: 

•  Routine  tasks 

•  Tasks  involving  lifting  or  exertion 


Analyst 


Scenario  # 


Job: 


Start  time: 


Performance  Measures 


How  is  your  performance  measured? 
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Part  I  -  Work  Content  (Description  of  Tasks  Performed)  (Cont’d) 


Maintenance  and  Inspection  Task  Key  List 


66.  abrading 
68.  bolting/screwing 

70.  chipping 

71.  cleaning  by  hand 

72.  cleaning  with  high  pressure  equipment 

73.  coating/immersing 

76.  crimping 

77.  cutting/shearing 

79.  drilling 

80.  driving  (vehicles) 

81.  excavating 

83.  flame  cutting/arc  cutting 

84.  folding/fitting 

85.  glueing/laminating 

86.  grinding/buffing/polishing 

87.  hammering 

88.  lifting 

90.  lubricating 

91.  machining 

92.  masoning 

93.  melting  ^ 


94.  molding 

97.  nailing 

98.  opening/closing  heavy  doors 

100.  painting/spray  painting 

101.  paving 

102.  pumping  (by  hand) 

103.  rivedng/bucking 

104.  sanding 

105.  sawing 

107.  sewing 

108.  soldering^razing 

110.  stripping/depainting  by  hand 

111.  stripping/depainting  mechanically 

113.  turning  valves 

114.  tying/twisting/wrapping 
116.  welding 

118.  wiring 

119.  wrenching/ratcheting 
(Write  in  others) 

121.  _ 

122.  _ 
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Part  II  -  Checklist,  Shoulder  /  Neck 
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Head/neck  bent  or  twistec 
(e.g„  craning  neck  looking 

into  light  spaces) _ 

Task  Scores  = 

(column  total) _ 
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(e.g.,  holding  a  pair  of 
fingernail  clippers  closed) 
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Part  II  -  Checklist,  Back/Torso 
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Part  m  -  Environmental 


Environmental  Factors 


Strongly 

Disagree 

Neutral 

Agree 

Strongly 

Disagree 

Agree 

1 

2 

3 

4 

5 

30.  Restricted  space 

32.  Windy  conditions 

33.  Extreme  temperatures  heat  /  cold 

34.  Noise  or  distractions 

35.  Air  quality  concerns 

Environmental  Score  = 


Environmental  Rating 
Environmental  Score 


Low 

0-3 

Part  IV  -  Employee  Suggestion 


Ask  the  employee  for  any  suggestions  for  corrective  actions  that  they  may  have. 


Stop  time: 
Elapsed  time: 


ERGONOMIC  SUMMARY  REPORT 


Technician 


rintion:  Please  write  out  job  description. 


Transfer  scores  from  individual  scoring  sheets. 


Task  Scores 


Priority 
Score  by 
Body 
Region 


Task  Name: 


Task  Name: 


Task  Name: 


Task  Name: 


Priority 
Rating  by 
Body 
Region 


Add  across  High:  8+ 
Med:  4-7 

divide  by  #  *  . 

oftasksfor  Low:  0-3 


Select  the  highest  body 
region  score  for  each  task 
then  circle  below  for  High, 
Med,  Low 

Highest  Score 

High:  8+ 

Med:  4-7 

High 

Med 

Low:  0-3 

Low 

Highest  Score 

Highest  Score 

High 

High 

Med 

Med 

Low 

Low 

Environmental 

Rating 


DRAFT! 
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Appendix  F 


Level  I  -  Ergonomics  Assessment  for  Maintenance  and  Inspection  Work  Areas  Page  1 


Part  I  -  Work  Content  (Description  of  Tasks  Performed) 


For  this  section,  work  with  the  employee  to  obtain  a  basic  descr^on  of  the  types  of  tasks  that  make  up  the  job  being 
evaluated.  Use  the  Maintenance  and  Inspection  Task  Key  List  as  a  reference.  If  the  en5>loyee  mentions  tasks  which  are  not 
included  on  the  Task  Key  list,  write-in  the  additional  tasks. 

For  each  task  performed,  determine  the  approximate  task  frequency  using  the  following 
proportions  of  job  time; 

•  Routine: 

-  >  50  %  (High):  The  total  percentage  the  job  spent  performing  the  task  is  greater 
than  50%. 

—  10-50  %  (Moderate):  The  total  percentage  of  the  job  spent  performing  the  task 
is  between  10  and  50%. 

•  <10%  (Low):  The  total  percentage  of  the  job  spent  performing  the  task  is  less  than 

10%. 

•  Non-routine:  Ihe  task  is  not  performed  during  the  normal  course  of  work  or  is 
performed  under  special  circumstances. 

For  each  ta-sk,  dieck  the  most  appropriate  circle  in  the  Work  Coirtent  Matrix  below. 

If  lifting  or  High  force  exertions  occur  in  the  task,  indicate  by  checldng  the  ^)propriate  circle. 


Analyst 


Scenario  # 


Job: 


Start  time: 


WORK  CONTENT  MATRIX 


Task 

Liftino  / 
Exertion 

Task  Freauencv 
(Check  one) 

1  Roatine  I 

Non*Roatiiie 

(Low) 

0-9% 

(Moderate) 

lO-SOVo. 

(High) 

51-100% 

1. 

o 

O 

2. 

o 

O 

3. 

o 

O 

4. 

o 

O 

5. 

o 

O 

■■M 

6. 

o 

O 

occur 


=  Critical  tasks  are  indicated  by  the  shaded  boxes  in  the  Work  Content  Matrix.  Critical  tasks  are  tasks  which 
greater  than  10%  of  the  job  time  or  which  involve  lifting  or  high  forces. 

ONLY  COMPLETE  THE  CHECKLIST  FOR  CRITICAL  TASKS, 


Performance  Measures 


How  is  your  performance  measured? 


Level  I  -  Ergonomics  Assessment  for  Maintenance  and  Inspection  Work  Areas  Page  2 


Part  I  -  Work  Content  (Description  of  Tasks  Performed)  (Cont) 


Maintenance  and  Inspection  Task  Key  List 


94.  molding 

97.  nailing 

98.  opening/dosing  heavy  doois 


66.  abrading 
68.  bolting/screwing 

70.  chipping 

71.  cleaning  hand 

72.  cleaning  with  high  pressure  equ^ment 

73.  coating/inmieising 

76.  cringing 

77.  cutting/sheaiing 

79.  drilling 

80.  driving  (vehicles) 

81.  excavating 

83.  flame  cutting(arc  cutting 

84.  folding^tting 

85.  glueing/laniinating 

86.  gtinding/bufBng^lishing 

87.  hammering 

88.  lifting 

90.  lubricating 

91.  machining 

92.  masoning 

93.  melting 


100.  painting/spray  painting 

101.  paving 

102.  purqsing  (by  hand) 

103.  liveting/baddng 

104.  sanding 

105.  sawing 

107.  sewing 

108.  soldering/brazing 

110.  stripping(depaintingly  hand 

111.  sti^ing/depainting  mechanically 

113.  tumingvalves 

114.  ^g/twistingfwrapping 
116.  welding 

118.  wiring 

119.  wrenching(ratcheting 
(Write  in  others) 

121 _ 

122 _ _ 
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Pa  rt  II  -  Checklist,  55houlder  /  Neck 
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Task  Scores 
(column  total] 


Level  I  Ergonomics  Assessment  for  Maintenance  and  Inspection  Work  Areas  Page  4 

Pa  rt  II  -  Checklist,  HandsAV rists/Arms 


Transfer  these  column  totals  to  the  next  page , 


Level  1  Ergonomics  Assessment  for  Maintenance  and  Inspection  Work  Areas  Page  5 

Part  II  -  Checklist,  HandsAVrists/Arms  (Conf  d) 


&  § 

«  s 

1  Is 
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2  g| 
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Part  II  -  Checklist,  Legs/Feet 


u.  & 


•SJ 


fi 

P 


3 

o 

a 


Vi 

tn 

H 


w 


Level  I  Ergonomics  Assessment  for  Maintenance  and  Inspection  Work  Areas  Page  9 
Part  III  "•  Environmental 


Environmental  Factors 


Strongly 

Disagree 

Disagree 

Nentral 

Agree 

Stron^y 

Agree 

1 

2 

3 

4 

5 

29.  Restricted  q)ace 

0 

0 

0 

1 

4 

30.  Windy  conditions 

0 

0 

0 

1 

4 

31.  Extreme  tenq>eratures  heat  /  cold 

0 

0 

0 

1 

4 

32.  Noise  or  distractions 

0 

0 

0 

1 

4 

33.  Air  quali^  concerns 

0 

0 

0 

1 

4 

Environmental  Score  = 


Environmental  Rating 
Environmental  Score 


Part  rV  -  Employee  Suggestion 


Ask  the  employee  for  any  suggestions  for  corrective  actions  that  they  may  have. 


Stop  time:. 
Elapsed  time: 


ERGONOMIC  SUMMARY  REPORT: 

Technician  ■  _ _ _ — 


Scenario  number 


Start  Time 


Scenario  name 


1.  Job  Description:  Please  write  out  job  description. 


2.  Scoring  SummairiTransfer  scores  from  individual  scoring  sheets. 


Body  Region 


Task  Scores 


Priority 
Score  by 
Body 
Region 


Task  Name: 


row  and 
divide  by  U 
of  tasks  for 
averaee 


Priority 
Rating  by 
Body 
Region 


High: 

Med: 

Low: 

8+ 

4-7 

0-3 

High 

Med 

Low 

High 

Med 

Low 

High 

Med 

Low 

Select  the  highest  body 
region  score  for  each  task 
then  circle  below  for 
High,  Med,  Low 


High:  8+ 

Med:  4-7 

Low:  0-3 _ 


Highest  Score 

Highest  Score 

Highest  Score 

Highest  Score 

High 

High 

High 

High 

Med 

Med 

Med 

Med 

Low 

Low 

Low 

Low 

Environmental 

Rating 


First  Priority  Task 


Select  the  highest  task  score. 
Write  the  task  name. 

Task  Name _ 

Select  the  two  highest  body  region  scores 
within  the  task. 


Second  Priority  Task 


Overall 


First  Highest 
Body  Region 

Score _ 

Body  Region^ 


Second  Highest 
Body  Region 

Score _ 

Body  Region _ 


Select  the  next  highest  task  score. 

Write  the  task  name. 

Task  Name 

Select  the  two  highest  body  region  scores 

within  the  task. 

First  Highest 

Second  Highest 

Body  Region 

Body  Region 

Score 

Score 

Body  Region 

Body  Region 

Priority  Rating 


DRAFTS 
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Level  I  -  Ergonomics  Assessment  for  Maintenance  and  Inspection  Work  Areas  Page  1 


Part  I  -  Work  Content  (Description  of  Tasks  Performed) 


For  this  section,  work  wiA  the  employee  to  obtain  a  basic  description  of  the  types  of  tasks  that  make  up  the  job  being 
evaluated.  Use  the  Maintenance  and  Inspection  Task  Key  List  (on  the  next  page)  as  a  reference.  If  the  employee  mentions 
tacifc  which  are  not  included  on  the  Task  Key  List,  write-in  the  additional  tasks. 


For  each  task  performed,  determine  the  approximate  task  frequency  using  the  following 
proportions  of  job  time: 

*  •  >  50  %  (High):  The  total  percentage  of  work  time  spent  performing  the  task  is 

greater  than  50%. 

•  •  10-50  %  (Moderate):  The  total  percentage  of  work  time  spent  performing  die  task 

is  between  10  and  50%. 

•  <  10  %  (Low):  The  total  percentage  of  work  time  spent  performing  the  task  is  less 

dian  10%. 

For  each  check  the  most  appropriate  circle  in  the  Work  Content  Matrix  below. 

If  lifting  or  high  force  exertions  occur  in  the  task,  indicate  by  checking  the  appropriate  circle. 


Analyst 


Scenario  # 


Job: 


Start  time: 


WORK  CONTENT  MATRIX 


Task 

Uftinoi 

Exertion 
Occur  in  Jack 

Task  Frequency 
(Check  one) 

(Low) 

0-9% 

(Moderate) 

10-50%. 

(High) 

51-100% 

1. 

O 

1  are  indicated  by  the  shaded  boxes  in  the  Work  Content  Matrix.  Critical  tasks  are  tasks  which 
occur  greater  than  10%  of  the  job  time  or  which  involve  lifting  or  high  forces. 

ONI.Y  COMPLETE  THE  CHECKLIST  FOR  CRITICAL  TASKS. 

T.nw  FRF.nTTFNf!Y  TASKS  WITH  LIFTING  OR  EXERTION  ARE  SCORED  AS  MODERATE  FREQUENCY. 


Performance  Measures 


How  is  your  performance  measured? 


Part  I  -  Work  Content  (Description  of  Tasks  Performed)  (Cont.) 


Maintenance  and  Inspection  Task  Key  List 


66.  abrading 

68.  bolting/screwing 

70.  chipping 

71.  cleaning  by  hand 

72.  cleaning  with  high  pressure  equipment 

73.  coatingdnunersing 

76.  crimping 

77.  cutting/shearing 

79.  drilling 

80.  driving  (vehicles) 

81.  excavating 

83.  flame  cutting/arc  cutting 

84.  folding/fitting 

85.  gluing/laminating 

86.  grinding/buffing/polishing 

87.  hammering 

88.  lifting 

90.  lubricating 

91.  machining 

92.  masoning 

93.  melting 

94.  molding 

95.  monitoring 

97.  nailing 

98.  opening/closing  heavy  doors 

100.  painting/spray  painting 


101.  paving 

102.  pumping  (by  hand) 

103.  riveting/bucking 

104.  sanding 

105.  sawing 

107.  sewing 

108.  soldering/brazing 

110.  stripping/depainting  by  hand 

111.  stripping/depainting  mechanically 

113.  turning  valves 

114.  tying/twisting/wrapping 
116.  welding 

118.  wiring 

119.  wrenching/ratcheting 

121.  assembly/disassembly  internal  component 

122.  assembly  repair  bench  work 

123.  computer  work 

124.  masking 

125.  mediablasting  glove  box 

126.  mediablasting-wholc  body 

127.  plying 

128.  solvent  cleaning 

129.  visual  inspection-machine  aided 

130.  visual  inspection-non  machine  aided 
(Write  in  others) 

131.  _ _ 

132 _ 


Level  I  Ergonomics  Assessment  for  Maintenance  and  Inspection  Work  Areas 
Part  n  -  Checklist,  Shoulder  /  Neck 


Pages 


Job  Factors 


For  each  Job  Factor,  select  the  q)propriate  Job  Factor  fircqucncy  score  using  the  following  guidelines: 

Frequently  (F):  Job  Factor  occurs  Sometimes  (S):  Job  Factor  occurs  Occasional  (O):  Job  Fac 
for  greater  than  50%  of  the  task  for  10-50%  of  the  task  for  less  than  10%  of  the  t 


Occasional  (O):  Job  Factor  occurs 
for  less  than  10%  of  the  task 
Critical  Tasks 


Never  (N):  Job  Factor  docs 
not  occur  or  does  not  apply 


Task  Name: 


Task  Name: 


Task  Name: 


Task  Frequency  I  Task  Frequency  |  Task  Frequency 


Moderate 

10-50% 


High 

51-100% 


Moderate 

10-50% 


High 

51-100% 


Moderate 

10-50% 


High 

51-100% 


1.  Reaching  repeated  readtmg  or 
caws  held  continuously  away  from 
body  while  unsupported _ 


Below  shoulder  level 
(arm  away  from  bo<fy) 


Above  shoulder  level 
(arm  >  90^  away  from  body) 


2.  Arm  forces: 

Repeated  arm  forces 

exceeding  10  lbs.  (e.g.  roughly 
equivalent  to  lifting  a  gallon  of 
ndlk)  or 

Holding/carrying  materials 
exceeding  25  lbs,  for  more  than 
three  steps 


3.  High  speedy  sudden  shoulder 
movements  (e,g,»  opening  a 
stuck  door,  pulling  and  yanking  on 
a  stuck  component  to  remove  it) 


A,  Head/neck  bent,  tilted,  or 
twisted  (>10®)  (e,g.,  craning 
neck  looking  into  tight  spaces) 


Task  Scores  = 
(column  total) 


F  S  0  N 

F  S  0  N 

110  0 

3  2  10 

F  S  0  N 

F  S  0  N 

3  2  10 

4  3  10 

F  S  O  N 

F  S  0  N 

2  10  0 

5  2  10 

F  S  0  N 

F  S  O  N 

2  10  0 

5  2  10 

F  S  0  N 

F  S  O  N 

3  2  10 

6  3  10 

F  S  0  N 

F  S  ON 

110  0 

3  2  10 

F  S  0  N 

F  S  0  N 

3  2  10 

4  3  10 

F  S  0  N 

F  S  0  N 

2  10  0 

5  2  10 

F  S  0  N 

F  S  0  N 

2  10  0 

5  2  10 

F  S  O  N 

F  S  O  N 

3  2  10 

6  3  10 

F  S  ON 

110  0 

F  S  0  N 

3  2  10 

F  S  O  N 

F  S  O  N 

3  2  10 

4  3  10 

F  S  O  N 

F  S  O  N 

2  10  0 

5  2  10 

F  S  0  N 

F  S  O  N 

2  10  0 

5  2  10 

F  S  O  N 

F  S  O  N 

3  2  10 

6  3  10 

Comments 


Level  I  Ergonomics  Assessment  for  Maintenance  and  Inspection  Work  Areas 
Part  n  -  Checklist,  HandsAVrists/Arms 


Page  4 


Job  Factors 


For  each  Job  Factor,  select  the  appropriate  Job  Factor  frequency  score  using  the  following  guidelines: 

FreouentlytF):  Job  Factor  occurs  Sometimes  (S):  Job  Factor  occurs  Occasional  ((^  Job  Factor  occurs 

fnr  than  10^^  of  the  task 


for  greater  than  50%  of  the  task  for  10-50%  of  the  task 


5.  Bent  wrists/repeated 
wrist  movements  (>10°  in 
any  direction)  or  repeated 
forearm  rotation  (e.g., 

turning  a  screw  driver,  Allen 
wrench) 


6.  Repeated  manipulations 
with  fingers  (e.g,  repetitive 
cannier  keying  tasks,  removing 
small  screws,  electrical  wiring 
tasks) _ ■ 


7.  Hyperextension  of 
finger/thumb  fe.g.,  using 
pliers  with  a  wide  handle  span) 
or  repeated  single  finger 
activation  (e.g,  single  finger 
triggers  on  power  tools) 


8.  Hand/grip  forces 

tinvertip  force:  >  2  lb.  (e.g,  2  lb. 
is  rougfily  equal  to  holding 
fingernail  clippers  closed 
full  hand  force:  >  8  lb.  (e.g.,  8  lb. 
is  roughly  equal  to  holding  a  8  lb. 
tool  or  holding  a  gallon  of  milk) 


9.  High  speed  hand/wnst 
/arm  movements  (e.g.,  yank  i 
components  with  fingers,  using  \ 

the  hand  as  a  hammer)  QL 

Vibration,  impact,  or 
torque  to  the  hand  (e.g., 

using  a  nail  gun  or  other  power 
tools  and  equipment) _ 


10.  Exposure  to  hard  edges 
(e.g.,  tool  handle  or  work  area 
presses  into  fingers  or  palm  of 
hands)  _ 


1 1.  Hands  and  fingers  exposed 
to  cold  temperatures  (e.g., 
working  outside  in  winter 
environment,  cold  exhaust  air  from 
tool  blows  on  hand/wrist)  _ 


Task  Scores* 

(column  total) _ 


Task  Name: 

Task  Frequency 

1  Moderate 

1  10-50% 

High 

51-100% 

FSON 

2  10  0 

FSON 

5  2  10 

FSON 
10  0  0 

FSON 

2  10  0 

FSON 
10  0  0 

FSON 

3  10  0 

FSON 

3  10  0 

FSON 

4  2  10 

for  less  than  10%  of  the  task 

Critical  Tasks 


Task  Name: 


Task  Frequency 


Moderate  High 
10-50%  51-100% 


FSON  |FSON 
2  10  0 


FSON 
10  0  0 


FSON 
10  0  0 


FSON 
2  10  0 


FSON 
3  10  0 


Task  Name: 

Task  Frequency 

Moderate 

10-50% 

High 

51-100% 

FSON 

2  10  0 

FSON 

5  2  10 

FSON 
10  0  0 

FSON 

2  10  0 

Never  (N):  Job  Factor  does 
not  occur  or  docs  not  iq)ply 


Comments 


FSON 
3  10  0 


FSON 
4  2  10 


FSON 
10  0  0 


FSON 
2  10  0 


Level  I  Ergonomics  Assessment  for  Maintenance  and  Inspection  Work  Areas 
Part  n  -  Checklist,  Back/Torso 


Pages 


Job  Factors 


For  each  Job  Factor,  select  the  appropriate  Job  Factor  frequency  score  using  the  following  guidelines: 

FreqoenaytF):  Job  Factor  occurs  Sometimes  (S):  Job  Factor  occurs  Occasional  (<^  Job  Factor  occurs 

for  meater  than  50%  of  the  task  for  10-50%  of  the  task  for  less  than  10%  of  the  task  i 

Critical  Tasks 


Task  Name: 


Task  Name:  I  Task  Name: 


Never  (N):  Job  Factor  docs 
not  occur  or  does  not  apply 


Task  Frequency  I  Task  Frequency 


12.  Repeated  forward  or  side¬ 
ways  bending  movements 
(>20®)  (c.g.  lifting  from  floor 
level)  _ 


13.  Twisting  ofthe  lower  back 
(e.g.  rushing  ^ile  lifting, 
pulling,  open  a  stuck  door) 


14.  High  speedy  sudden 
movements  with  the  back 


15.  Static,  awkward  back 

postures  {for  >10  sec  at  a  tune) 
While  Standing,  continuous 
*  leaning  forward  or  to  the  side 

(>2(P)  nr  While  seated, 

continuous  leaningforward 
(>20®)  or  poor  lower  back 
posture  (e.g„  poor  lower  back 
support,  no  support  for  feet) _ 


16.  Lifting  forces 


•  SO^IO  lb.  while  tgtright  w/ 
load  close  to  body  nc 


•  10-40  lb.  while  bending  or 

reaching 


•  >70  lb.  while  tqtright  w/ 

load  close  to  body  qc 


•  >40  lb.  while  bending  or 

reaching 


17.  Pushing  or  pulling 

force  >  50  lb.)  (e.g. 
pushing/puHing  a  full  two-drawer 
file  cabinet  across  a  carpeted 

flo^ _ _ 


18.  Whole  body  vibration 
througft  floor  surface  (e.g. 
operating  heavy  machinery) 


Task  Scores* 
(column  total) 


Moderate 

10-50% 


F  S  O  N 
2  10  0 


F  S  ON 
3  10  0 


F  S  O  N 
3  10  0 


High 

51-100% 


F  S  O  N 
3  2  10 


F  S  O  N 
4  2  10 


F  S  O  N 
4  2  10 


Moderate 

10-50% 


High 

51-100% 


F  S  O  N 

2  10  0 

F  S  0  N 

3  2  10 

F  S  O  N 

F  S  0  N 

3  10  0 

4  2  10 

F  S  O  N 

F  S  0  N 

3  10  0 

4  2  10 

F  S  0  N 

F  S  0  N 

2  10  0 

6  2  10 

Moderate 

10-50% 


F  S  O  N 
2  10  0 


F  S  O  N 
3  10  0 


High 

51-100% 


F  S  O  N 
3  2  10 


Comments 


Level  I  Ergonomics  Assessment  for  Maintenance  and  Inspection  Work  Areas 
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Part  11- Checklist,  Legs/Feet 


Job  Factors 


For  each  Job  Factor,  select  the  appropriate  Job  Factor  frequency  score  using  the  following  guidelines: 


rrcuuvuuj  - -  -  ^  ,  , 

for  greater  than  50%  of  the  task  for  10-50%  of  the  task 


for  less  than  10%  of  the  task  i 

Critical  Tasks 


Task  Name:  I  Task  Name: 


Never  (N):  Job  Factor  does 
not  occur  or  does  not  ^ply 
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Job  Factors 


For  each  Job  Factor,  select  the  ^propriate  Job  Factor  frequency  score  using  the  following  guidelines: 


Frequently  (F):  Job  Factor  occurs 
for  greater  than  50%  of  the  task 


Sometimes  (S);  Job  Factor  occurs 
for  10-50%  of  the  task 


Occasional  (O):  Job  Factor  occurs 
for  less  than  W/o  of  the  task 


Job  Factor 


23.  Difficult  to  see/light  levels 
too  low  /too  high. 

(e.g.,  see  detail) 


24.  Intensive  visual  tasks, 
staring  at  work  objects 
for  long  periods  (e.g.. 
inspection, 
troubleshooting) 


Task  Scores  B 
(column  total) 


Task  Name: 

Task  Name: 

Task  Name: 

Task  Frequency 

Task  Frequency 

Task  Frequency 

1  Moderate 

1  10-50% 

High 

51-100% 

Moderate 

10-50% 

High 

51-lOOVo 

Moderate 

10-50% 

High 

51-100*/* 

F  S  O  N 

2  10  0 

F  S  0  N 

3  2  10 

F  S  O  N 

2  10  0 

F  S  0  N 

3  2  10 

F  S  0  N 

2  10  0 

F  S  0  N 

3  2  10 

F  S  O  N 

2  10  0 

F  S  O  N 

3  2  10 

F  S  O  N 

2  10  0 

F  S  O  N 

3  2  10 

F  S  O  N 

2  10  0 

F  S  O  N 

3  2  10 

Never  (N):  Job  Factor  does 
not  occur  or  does  not  ^ply 


Comments 
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Environmental  Factors 


Strongly 

Disagree 


25.  Restricted  space 


26.  Windy  conditions 


27.  Extreme  temperatures  heat  /  cold 


28.  Noise  or  distractions 


29.  Air  quality  concerns 


30.  Displays/controls  are  difficult  to  understand 


Disagree 

2 

Neutral 

3 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

Agree 


Strongly 

Agree 


Environmental  Score  - 


Environmental  Rating  Low  - 

Environmental  Score  0*3 


Part  rV  -  Employee  Suggestion 


Stop  time:. 
Elapsed  time: 


ERGONOMIC  SUMMARY  REPORT 


Start  Time 


Technician 


Scenario  number 


Scenario  name 


rintion:  Please  write  out  job  description. 


Body  Region 


Transfer  scores  from  individual  scoring  sheets. 


Task  Scores 


Task  Name: 


Task  Name:  I  Task  Name:  I  Task  Name: 


Priority  Priority 

Score  by  Rating  by 
Body  Body 

Region  Region 


Add  across  High:  8+ 

Med:  4-7 
divide  by  #  -  ^  ^ 

of  tasks  for  Low.  0-3 


Select  the  highest  body 
region  score  for  each  task 
then  circle  below  for  High, 
Med,  Low 

Highest  Score 

Highest  Score 

Highest  Score 

Highest  Score 

High:  8+ 

Med:  4-7 

High 

Med 

High 

Med 

High 

Med 

High 

Med 

Low:  0-3 

Low 

Low 

Low 

Low 

Environmental 
Rating 


First  Priority  Task 


Select  the  highest  task  score. 
Write  die  task  name. 

Task  Name _ ___ 

Select  die  two  highest  body  region  scores 
within  die  task. 

First  Highest  Second  Highest 

Body  Region  Body  Region 

Score _  Score 

Body  Region _  Body  Region _ 


Second  Priority  Task 


Select  the  next  highest  task  score. 
Write  the  task  name. 

Task  Name _ 

Select  the  two  highest  body  region  scores 
within  the  task. 


First  Highest 
Body  Region 

Score _ 

Body  Region_ 


Second  Highest 
Body  Region 

Score _ 

Body  Region _ 


1  Overall  | 

Highest  Priority 
Score  by  Body 
Region 

Priority  Rating 

Score: 

High 

Body 

Med 

Reeion: 

Low 

1 


Customer  Customer 

POC  and  Phone: _  Office  Symbol: 


Date:. 


Division:  OEMO 


TECHNICAL  REPORT  SATISFACTION  SURVEY 
(AF  PREMIER  Program  Level  I  Ergonomics  Assessment  Guide- 

Maintenance/Inspection) 

This  survey  is  used  to  help  us  improve  our  service  to  you.  Your  answers  will  be  held  in  confidence  and  will  significantly  impact  how  we  allocate 
resources  to  meet  your  needs.  Upon  completion  of  your  review,  please  mail  or  fax  (DSN  240-2288/Comm  210-536-2288)  this  form  to  our  Quality 
Assurance  office  (OEPQ).  Thank  you  very  much! 


GRADING  SCALE:  EXTREMELY  SLIGHTLY  SLIGHTLY 

DISSATISFIED  DISSATISFIED  DISSATISFIED  SATISFIED  SATISFIED 

1  2  3  4  5 


EXTREMELY 
SATISFIED  N/A 

6 


FORMAT:  Is  the  report  understandable  and  well  organized? 

1 

2 

3 

4 

5 

6 

N/A 

CONTENT:  Does  it  provide  you  with  necessary  program 
implementation  guidelines  or  data  collection  tools? 

1 

2 

3 

■ 

5 

6 

N/A 

USEFULNESS:  Did  you  use  this  product  and,  if  so,  was  it  easy  to  use? 

1 

2 

3 

mm 

5 

6 

N/A 

QUALITY:  Are  you  satisfied  with  the  quality  of  this  product? 

1 

2 

3 

4 

5 

6 

N/A 

SUPPORT:  Are  you  satisfied  with  the  support  we  have  provided  on 
this  product? 

1 

2 

3 

■1 

5 

6 

N/A 

OVERALL:  Overall,  how  would  you  rate  this  product? 

1 

2 

3 

4 

5 

6 

N/A 

Comments/Suggestions:  Did  you  find  any  errors  or  omissions?  Will  you  continue  to  use  this  product?  Are  there  any  specifics 
of  this  product  you  would  like  to  discuss?  Are  there  other  services  you  would  like  provided  in  the  future?  (Use  back  of  page  if  more 
space  is  required.) 


OFFICIAL  BUSINESS 


(fold)- 


Return  to: 

Armstrong  Laboratory  /  OEPQ 
2402  E  Drive 

Brooks  AFB,TX  78235-5114 


